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Preface 



A large international conference on Advances in Intelligent Control and Computer 
Engineering was held in Hong Kong, March 17-19, 2010, under the auspices of 
the International MultiConference of Engineers and Computer Scientists (IMECS 
2010). The IMECS is organized by the International Association of Engineers 
(lAENG). lAENG is a non-profit international association for the engineers and 
the computer scientists, which was founded in 1968 and has been undergoing rapid 
expansions in recent years. The IMECS conferences have served as excellent venues 
for the engineering community to meet with each other and to exchange ideas. 
Moreover, IMECS continues to strike a balance between theoretical and application 
development. The conference committees have been formed with over two hundred 
and fifty members who are mainly research center heads, deans, department heads 
(chairs), professors, and research scientists from over thirty countries. The confer- 
ence participants are also truly international with a high level of representation from 
many countries. The responses for the conference have been excellent. In 2010, 
we received more than one thousand manuscripts, and after a thorough peer review 
process 56.26% of the papers were accepted (http://www.iaeng.org/IMECS2010). 

This volume contains 25 revised and extended research articles written by promi- 
nent researchers participating in the conference. Topics covered include artificial 
intelligence, control engineering, decision supporting systems, automated planning, 
automation systems, systems identification, modelling and simulation, communica- 
tion systems, signal processing, and industrial applications. The book offers the state 
of the art of tremendous advances in intelligent control and computer engineering 
and also serves as an excellent reference text for researchers and graduate students, 
working on intelligent control and computer engineering. 

Sio-Iong Ao 

Oscar Castillo 

Xu Huang 
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Intelligent Control of Reduced-Order Closed 
Quantum Computation Systems Using Neural 
Estimation and LMI Transformation 



Anas N. Al-Rabadi 



Abstract A new method of intelligent control for closed quantum computation 
time-independent systems is introduced. The introduced method uses recurrent su- 
pervised neural computing to identify certain parameters of the transformed system 
matrix [A]. Linear matrix inequality (LMI) is then used to determine the permuta- 
tion matrix [P] so that a complete system transformation {[B], [C], [D]} is achieved. 
The transformed model is then reduced using singular perturbation and state feed- 
back control is implemented to enhance system performance. In quantum computa- 
tion and mechanics, a closed system is an isolated system that can't exchange energy 
or matter with its environment and doesn't interact with other quantum systems. In 
contrast to an open quantum system, a closed quantum system obeys the unitary 
evolution and thus is information lossless that implies state reversibility. The exper- 
imental simulations show that the new hierarchical control simplifies the model of 
the quantum computing system and thus uses a simpler controller that produces the 
desired performance enhancement and system response. 

Keywords Linear matrix inequality ■ Model reduction • Quantum computation ■ 
Recurrent supervised neural computing ■ State feedback control system 



1 Introduction 

Due to the fact that current dense hardware implementations are heading towards 
the critical atomic threshold, quantum computing will rapidly occupy an increas- 
ingly important position in building nano-size, super-fast, and ultra-low power con- 
suming systems [1-3, 6, 8, 12]. Other motivations for implementing circuits and 
systems using quantum computing would include items such as: (1) power where 
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State Feedback Control System 



Model Reduction 



System Transformation: {[B], [C], [D]} 



LMI-Based Permutation Matrix: [P] 



Neural-Based State Transformation: [A] 



Time-Independent Quantum Computing System: {[A], [B], [C], [D]} 
Fig. 1 The introduced control methodology utilized for closed quantum computing systems 



the internal computations in quantum computing systems consume no power and 
only power is consumed when reading and writing operations are performed [1,6, 
8, 12]; (2) size where, at the atomic dimension, quantum mechanical effects have to 
be accounted for; and (3) speed where if the properties of superposition and entan- 
glement of quantum mechanics can be usefully employed in the design of circuits 
and systems, significant computational speed enhancements can be expected [1,6, 
12]. Figure 1 illustrates the layer layout of the introduced closed-system quantum 
computing control methodology. 



2 Fundamentals 

This section presents important background on quantum computing systems, super- 
vised neural networks, linear matrix inequality, and model order reduction that will 
be used later in Sects. 3, 4 and 5. 



2.1 Quantum Computation 

Quantum computing is an efficient method of computation that uses the dynamic 
process which is governed by the Schrodinger equation [1, 6, 12]. The one- 
dimensional time-dependent Schrodinger equation (TDSE) is as follows [1, 5, 6, 
12]: 

(h/lTtf d^li^) , ^^, ,^ d\ir) 

^ ^^ + V\i/)^i(h/27T)—— (1) 

Zm ax'^ at 

or H\,l,)^i(h/27t)^ (2) 

at 

where h is Planck constant (6.626 ■ 10"^"^ J ■ s = 4.136 ■ 10"^^ eV • s), V(x, t) is the 
applied potential, m is the particle mass, i is the imaginary number, \ifix., t)) is the 
quantum state, H is the Hamiltonian operator where H — —[{h/2jt)^/2 m] V^ + V , 
and V^ is the Laplacian operator. 
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A general solution to the TDSE is the expansion of a stationary (i.e., time- 
independent for spatial) basis functions (i.e., eigen states) Ue(r) using time- 
dependent (i.e., temporal) expansion coefficients Ce{t) as follows; 

n 

*(?,f)=^Q(f)M,(?) 

e=0 

The expansion coefficients Ce(t) are a scaled complex exponentials as follows: 

Q(r) = A:,e-'wfe)' 

where Eg, are the energy levels. In quantum computing, the time-independent 
Schrodinger equation (TISE) is normally used [1, 12]: 

^'^ = 7]:7r\2(^-'^)'^ (3) 

(h/zjiy 

where the solution |i/r) is an expansion over orthogonal basis states |0/} defined in 
a linear complex vector space called Hilbert space H as: 

|iA) = ^Q|0,) (4) 

where the coefficients c, are called probability amplitudes and Ic, p is the probability 
that the quantum state \\[r) will collapse into the (eigen) state | (/),). The probability 
is equal to the inner product | ((/>,■ |iA)|~, with the unitary condition ^ |c,p = 1. In 
quantum computing, a linear and unitary operator 3 is used to transform an input 
vector of quantum bits (qubits) into an output vector of qubits. In the two- valued 
quantum computing, the qubit is a vector of bits which is defined as follows [1, 12]: 



qubito = |0) = 



qubiti = |l) = 



(5) 



A two-valued quantum state \-^) is a superposition of quantum basis states 10,- ). 
Thus, for the orthonormal computational basis states {|0), |1)}, one has the following 
quantum state: 

\if)^a\0)+p\l) (6) 

where aa* — \a\- — pQ = the probability of having state |i/f) in state |0), /6y6* = 
lySp — pi = the probability of having state \xj/) in state |1), and \a\^ + \P\^ — 1. The 
calculation in quantum computing for multiple systems follows the tensor product 
(0). For example, given the quantum states \\[ri) and |i/^2>, one has: 

\i^lf2}^\fl)®\f2} 

= (ai|0)+/Ji|l))®(a2|0)+;62|l» 

= aiQ:2|00)+ai,62|01)+/Sia2|10)+/SiyS2|ll> (7) 

A physical system (e.g., the hydrogen atom) that is described by the following 
equation: 

|i/f) —ci ISpinup) + C2|Spindown) (8) 
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can be used to physically implement a two-valued quantum computing. Another 
common alternative form of Eq. 8 is as follows: 



W) =ci 



-C2 



(9) 



Many-valued quantum computing can also be performed. For the three-valued 
case, the qubit becomes a 3D vector quantum discrete digit (qudit), and in general, 
for an m -valued quantum computing the qudit is of dimension "many" [1, 12]. For 
example, one has for the 3-state case, the following qudits: 



qudito = |0) 



1 



LOJ 



quditi^ll) 



qudit2 = |2} 



(10) 



A three-valued quantum state is a superposition of three quantum orthonormal basis 
states (vectors). Thus, for the orthonormal computational basis states {|0), |1}, |2)}, 
one has the following quantum state: 

\if)^a\0)+fi\\) + y\2) 



where aa* — \a\ — pQ = the probability of having state \-^) in state |0), 



'2 = 1. 



; the 



|y6p = pi = the probability of having state |i/r) in state |1) 
probability of having state {-^r) in state |2), and |ap + |yS|2 + |y| 

The calculation in quantum computing for m -valued multiple systems follow 
the tensor product in a manner similar to the one demonstrated for the higher- 
dimensional qubit in the two-valued quantum computing. Several quantum comput- 
ing systems were used to implement quantum gates from which complete quantum 
circuits and systems were constructed [1, 6, 12], where several of the two-valued 
and m -valued quantum circuit implementations use the two- valued and m -valued 
quantum Swap-hased and A^or-based gates [1, 12]. 

In general, for an w -valued logic, a quantum state is a superposition of m quan- 
tum orthonormal basis states (i.e., vectors). Thus, for the orthonormal computational 
basis states {|0), |1), . . . , |m — 1)}, one has the quantum state: 



IV^) = ^Ck\q)k 



(11) 



A.-0 



Where I:L"o^^< 



Em — 1 I 1 2 



1. The calculation in quantum computing for 
m -valued multiple systems is done similar to the case for the two- valued system. 

In quantum mechanical systems, a closed system is an isolated system that 
doesn't exchange energy or matter with its environment (i.e., doesn't dissipate 
power) and doesn't interact with other quantum systems. While an open quantum 
system interacts with its environment and thus dissipates power which results in a 
non-unitary evolution producing information loss, a closed quantum system doesn't 
exchange energy or matter with its environment and therefore doesn't dissipate 
power which results in a unitary evolution (i.e., unitary matrix) and thus it is in- 
formation lossless. 
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Fig. 2 The utilized second 
order recurrent neural 
network architecture, where 
the estimated matrices are 
given by 



and W = [[Ad] [Brf]U)] 



Outputs 

yik) 



System state: 
internal input 



Bi] Neuron 




System 
external 
input 



2.2 Recurrent Supervised Neural Computations 



The supervised recurrent neural network which is used for the estimation in this 
research is based on an approximation of the method of steepest descent [2, 9]. The 
network tries to match the output of certain neurons to the desired values of the 
system output at a specific instant of time. Figure 2 shows a network consisting of 
a total of N neurons with M external input connections for a 2"'^ order system with 
two neurons and one external input, where the variable g{k) denotes the (M x 1) 
external input vector which is applied to the network at discrete time k and the 
variable y(k + 1) denotes the corresponding (A'^ x 1) vector of individual neuron 
outputs produced one step later at time (A: + 1 ) . 

The derivation of the recurrent algorithm can be started by using dj (k) to denote 
the desired (i.e., target) response of neuron j at time k, and g{k) to denote the set 
of neurons that are chosen to provide externally reachable outputs. A time- varying 
(A^ X 1) error vector e(k) is defined whose y* element is given by the following 
relationship: 

dj(k)-yj(k), ifjeg(k) 
0, otherwise 



ej(k). 



The objective is to minimize the cost function iitotai which is obtained by £totai = 
^lcE(k) where E{k) — j^ygj-e^C^)- The dynamical system is described by the 

following triply indexed set of variables (tt^^): 

dyjik) 



'ml 



(k). 



dw,„((k) 

where for every time step k and all appropriate j, m and £, system dynamics are 
controlled by: 



,(k+l)^^{vj(k)) 



y^ Wji(k)jT'^^(k) + S,njUi(k) 



iep 
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with TT^^^CO) — 0. The values of Jt^^^ik) and the error signal ej{k) are used to com- 
pute the corresponding weight changes with a learning rate (r]) : 

Aw,ndk)^riJ2e.i(k)^L('') (12) 

Using the weight changes, the updated weight Wmiik + 1) is calculated as: 

w,„i(k+\)^Wmdk) + I^Wmi(k) (13) 

and repeating this computation procedure provides the minimization of the cost 
function and the objective is therefore achieved. 



2.3 Transformation via Linear Matrix Inequality 

In this sub-section, the detailed illustration of system transformation using LMI 
optimization will be presented [2]. Consider the system: 

kit) ^ Ax{t) + Bu{t) (14) 

y{t)^Cx{t) + Du{t) (15) 

In order to determine the transformed [A] matrix, which is [A], the discrete zero 
input response is obtained. This is achieved by providing the system with some 
initial state values and setting the system input to zero (i.e., u(k) — 0). Hence, the 
discrete system of Eqs. 14, 15, with the initial condition x (0) — xq, becomes: 

x{k + l)^Adx{k) (16) 

y(k)^x{k) (17) 

We need x (k) as a neural network target to train the network to obtain the needed 
parameters in [A^] such that the system output will be the same for [A^] and [A^]. 
Hence, simulating this system provides the state response corresponding to their 
initial values with only the [A^] matrix is being used. Once the input-output data is 
obtained, transforming the [A^] matrix is achieved using the neural network train- 
ing, as will be explained in Sect. 3. The estimated transformed [Aj] matrix is then 
converted back into the continuous form which yields: 



A = 



A„ 



(18) 



Having the [A] and [A] matrices, the permutation [P] matrix is determined using 
the LMI optimization technique [2, 4] as will be illustrated in later sections. The 
complete system transformation can be achieved by assuming that x — P~^x and 
then the system of Eqs. 14, 15 can be re-written as follows: 

Pxit) = APx(t) + Bu{t), y(t) = CPx{t) + Du{t) 
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where {y(t) — y(t)). Pre-multiplying the first equation above by [P~M> one obtains 
{P-^ Px(t) ^ P-^ APx(t) + P-^ Buit), y(t) ^ C Px(t) + Du(t)} v/hichyields the 
following transformed model: 

(19) 
(20) 







i(0 = 


Ax{t) + Bu{t) 






y(t)^ 


Cx(t) + bi 


uit) 


where the transformed 


system 


matrices are given 


by: 






A 


^p-^AP 








B 


^P-^B 








C 


^CP 








b 


= £) 





(21) 

(22) 

(23) 

(24) 

Transforming the system matrix [A] into the form shown in Eq. 1 8 can be achieved 
based on the property of matrix reducability [2, 10]. 



2.4 Singular Perturbation for Model Order Reduction 

Linear time-invariant models of many systems have fast and slow dynamics which 
is referred to as singularly perturbed systems [2, 11]. Neglecting the fast dynamics 
of a singularly perturbed system provides a reduced slow model leading to simpler 
controllers based on the reduced model information [2, 11]. For reduced system 
formulation, consider the following singularly perturbed system: 

x(t) = Anx{t) + An^{t) + BiM{t), a:(0) = xo (25) 

e|(0 = A2ix(0 + A22t(0 + fi2M(0, f(0) = |o (26) 

y{t)^Cix(t) + C2Ht) (27) 

where x e Jit™' and ^ e ilt'"^ are the slow and fast state variables, respectively, u e 
9{"i and y e ilt"^ are the input and output vectors, respectively, {[A„], [B,], [C,]} 
are constant matrices of appropriate dimensions with / e {1,2}, and e is a small 
positive constant. The singularly perturbed system in Eqs. 25, 26, 27 is simplified 
for e = O. By doing the above step, one neglects the system fast dynamics assuming 
that the state variables ^ have reached the quasi-steady state. Setting e = in Eq. 26 
and assuming [A22] is nonsingular, produces: 

^(0 = -A-^A2iXr(t) - A-^Biuit) (28) 

where the index r denotes remained (or reduced) model. By substituting Eq. 28 into 
Eqs. 25, 26, 27, one obtains the following reduced order model: 

i,(t) = ArXrit) + B,.u(t) (29) 

y(t)^CrxAt) + DMt) (30) 

for {Ar = All - Ai2A22'A21, 5, = Bl - Ai2A22'52, Cr^Cl- C2A~^A21,Dr = 
-C2A-'B2}. 
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3 Neural Estimation with Linear Matrix Inequality-Based 
Transformation for Closed Reduced-Order Quantum 
Computation Systems 

In this work, it is our objective to search for a similarity transformation that can be 
utilized within the context of closed time-independent quantum computing systems 
to decouple a pre-selected eigenvalue set from the system matrix [A] . To achieve this 
objective, training the neural network to estimate the transformed discrete system 
matrix [A^] is performed [2]. For the system of Eqs. 25, 26, 27, the discrete model 
of the quantum computing system is obtained as: 



yik) = Cdxik) + Ddu(k) 
The estimated discrete model of Eqs. 31, 32 can be re-written as: 



xi(k+l) 
xiik+l) 



All 
A21 



An 

A22 



xi(k) 

X2(k) 



fill 
B21 



u(k) 



y(k) = 



xi(k) 

X2{k) 



(31) 
(32) 



(33) 
(34) 



where k is the time index, and the matrix elements of Eqs. 33, 34 were shown in 
Fig. 2. The recurrent neural network that was presented in Sect. 2.2 can be sum- 
marized by defining A as the set of indices (/) for which gi{k)is an external input, 
which is one external input in the quantum computing system, and by defining fi 
as the set of indices (;) for which yi(k) is an internal input (or a neuron output), 
which is two internal inputs (i.e., two system states) in the quantum computing sys- 
tem. Also, we define m; (k) as the combination of the internal and external inputs for 
which i e /J U A. By using this setting, training the network depends on the internal 
activity of each neuron which is given by the following equation: 






(35) 



where Wji is the weight representing an element in the system matrix or input matrix 
for j e P and i e fiUA such that W — [ [Ad] [Bd] J • At the next time step (k + 1), 
the output (i.e., internal input) of the neuron j is computed by passing the activity 
through the nonlinearity 0(.) as follows: 



Xjik+l)^ip{vj{k)) 



(36) 



With these equations, based on an approximation of the method of steepest descent, 
the network estimates the system matrix [A^;] as was shown in Eq. 16 for zero input 
response. That is, an error can be obtained by matching a true state output with a 
neuron output as follows: 



ej (k) — Xj (k) — Xj (k) 
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The objective is to minimize the cost function Zstotai — ^k E(k) where E{k) — 
2 ^ g e^ (k) and g denotes the set of indices j for the output of the neuron struc- 
ture. This cost function is minimized by estimating the instantaneous gradient of 
E {k) with respect to the weight matrix [W] and then updating [W] in the negative 
direction of this gradient. In detailed steps, this may be proceeded as follows: 

- Initialize the weights [W] by a set of uniformly distributed random numbers. 
Starting at the instant k — 0, use Eqs. 35, 36 to compute the output values of the 
N neurons (where N — fi). 

- For each time step k and all y e p, m e /S, and £ e fiU A, compute the dynamics 
of the system governed by the triply indexed set of variables: 



7r,li(k+l)^<pivj(k)) 






^ik) + S,„jUi(k) 



with initial conditions :7r^^ (0) —OandS^i is given by (dw jiik)/ dw,„i(k)), which 
is equal to "1" only when j — m and ; — i otherwise it is "0". Note that for the 
special case of a sigmoidal nonlinearity in the form of a logistic function, the 
derivative (p{-) is given by (p{Vj(k)) = y j{k + 1)[1 — yj{k + 1)]. 
Compute the weight changes correspond to the error and system dynamics: 



Awmiik) ^r]'^ejik)7T^i,{k) 



J eg 



- Update the weights in accordance with: 

w„,t(k + 1) = w„,i(k) + Aw,„i(k) 

- Repeat the computation until the desired estimation is achieved. 



(37) 



(38) 



As was illustrated in Eqs. 16, 17, for the purpose of estimating only the transformed 
system matrix [A], the training is based on the zero input response. Once the train- 
ing is complete, the obtained weight matrix [W] is the discrete estimated trans- 
formed system matrix. Transforming the estimated system back to the continuous 
form yields the desired continuous transformed system matrix [A] . Using the LMI 
optimization technique that was illustrated in Sect. 2.3, the permutation matrix [P] 
is determined. Hence, a complete system transformation, as was shown in Eqs. 19, 
20, is achieved. To perform the order reduction, the system in Eqs. 19, 20 are written 



Xr(t) 
Xo(t) 

yrit)' 
yo(t) 



A, 





Ac 
Ao 



[Cr Co] 



Xr{t) 
Xoit) 

Xrit) 
Xo(t) 



+ 



Br 
Bo 

Dr' 
Do 



u{t) 



u(t) 



(39) 



(40) 



where the system transformation enables us to decouple the original system into 
retained (r) and omitted (o) eigenvalues. The retained eigenvalues are the dominant 
eigenvalues that produce slow dynamics and the omitted eigenvalues are the non- 
dominant eigenvalues that produce fast dynamics. Equation 39 can be re-written as 

{x,-(t) = ArXr(t) + AcXo(t) + BrUit), Xo(t) = Ao^oiO + BoU(t)}. 
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The coupling term A^Xoit) maybe compensated for by solving for xdt) in the 
second equation above by setting Xo{t) to zero using the singular perturbation 
method (by setting e = 0). Doing so, the following is obtained: 

Xoit)^-A-^BMt) (41) 

Using Xo(t), we get the reduced model given by: 

i,(f) = ArXrit) + [-AcA-^Bo + Br]u(t) (42) 

y(t) = C,.x,{t) + [-CoA-^B„ + D]u(t) (43) 

Therefore, the overall reduced order model is: 

Xr(t)^AorXr(t) + BorU(t) (44) 

y(t) = CorXr{t) + DorU(t) (45) 

where the details of the overall reduced matrices {[A^r], [Bor], [Cor], [Dor]} are 
shown in Eqs. 42, 43. 



4 JVIodel Order Reduction of the Quantum Computation 
Systems Using Neural Estimation and Linear IVIatrix 
Inequality Transformation 

Let us implement the time-independent quantum computing closed-system using the 
particle in finite-walled box potential V for the general case of m -valued quantum 
computing in which the resulting distinct energy states are used as the orthonormal 
basis states [2] . The dynamical TISE of the one-dimensional particle in finite-walled 
box potential V is expressed as follows: 

92*1/ 2m 

— T + t(£ - V)* = 

which also can be re-written as |-f^ — j^(V — E)^', where m is the particle mass, 
and h = (h/ljr) is the reduced Planck constant (which is also called the Dirac con- 
stant) = 1.055 ■ lO"^'* J ■ s = 6.582 ■ 10"'^ eV • s. Thus, for {xi = ^I/, X2 = ff , xj = 

X2, Xj = y't)' the state space model of the time-independent closed quantum com- 
puting system is given as: 

1 



2m(V-E) p) 



«)Ko)" ""' 



> = (1 0)Pj+(O)« («) 

For simulation reasons, Eqs. 46, 47 can also be re-written equivalently as: 
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2m(E-V) 





-X2 
XI 



^ = (0 1) 



-X2 
XI 



(0)m 



(48) 



(49) 



Also, for conducting the simulations, one may often need to scale the system 
Eq. 48 without changing the system dynamics. Thus, by scaling both sides of Eq. 48 
by a scaling factor a, the following set of equations is obtained: 







2m(E-V) 





-X2 
X\ 



^ = (0 1) 



-X2 
Xl 



+ (0)M 



Therefore, one obtains the following set of quantum system matrices: 



A = fl 






-1 


2m(E-V) 




B = 


"0" 
_0_ 






C = [0 1] 




£) = 


[0 


] 







(50) 
(51) 

(52) 

(53) 

(54) 
(55) 

The specifications of the system matrix in Eq. 52 for the particle in finite-walled box 
are determined by (1) potential box width L (in nanometer), (2) particle mass m, 
and (3) the potential value V (i.e., potential height in electron Volt). As an example, 
consider the particle in a finite-walled potential with specifications of (E — V) = 
88 MeV and a very light particle with a particle mass of N — 10~^^ of the electron 
mass (where the electron mass mg = 9.109 ■ 10~^^ g — 5.684 ■ 10~'^ eV/(m/s)^). 
This system was discretized using the sampling rate Ts — 0.005 second and sim- 
ulated for a zero input. Hence, based on the obtained simulated output data and 
by using NN to estimate the subsystem matrix [A^] of Eq. 18 with a learning rate 
r] = 0.015, the transformed system matrix [A] was obtained where [A^] is set to pro- 
vide the dominant eigenvalues (i.e., slow dynamics) and [A^] is set to provide the 
non-dominant eigenvalues (i.e., fast dynamics) of the original system. Thus, when 
training the system, the second state Xo(t) of the transformed model in Eq. 39 is un- 
changed due to the restriction of [ Ag] seen in [A] . This may lead to an undesired 
starting of the system response, but fast system overall convergence. 

Using [A] along with [A], the LMI is implemented to obtain {[B], [C], [D]} 
which makes a complete model transformation. Then, by using singular perturbation 
for model reduction, the reduced order model is obtained. Thus, by implementing 
the previously stated system specifications and by using the squared reduced Planck 
constant of h^ — 43.324 ■ 1G~^^ (eV • s)^, one obtains the following scaled system 
matrix from Eq. 52: 
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Fig. 3 (Color online) 
Input-to-output quantum 
computing .system step 
responses: full-order system 
model {solid blue line), 
transformed reduced-order 
model (dashed black line), 
and non-transformed 
reduced-order model (dashed 
red line) 
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Accordingly, the eigenvalues were found to be {—5.0399, —10.9601}. For a step 
input, simulating the original and transformed reduced order models along with the 
non-transformed reduced order model produced the results shown in Fig. 3. 



5 The Design of State Feedback Controller for the 
Reduced- Order Closed Quantum Models 



ind , 



In this research, since the closed quantum computing system is a 2" order system 
reduced to a P' order, we will investigate the system stability and enhancing perfor- 
mance by implementing the simple method of the 5 -domain pole replacement [2, 7]. 
For the reduced model in Eqs. 44, 45, a state feedback controller can be designed. 
For example, this can be achieved by replacing the system eigenvalues with new 
faster eigenvalues. Hence, let the control input be: 



u{t) — -Kxrit) +r(t) 



(56) 



where K is to be designed based on the desired system eigenvalues. 

Replacing the control input u(t) in Eqs. 44, 45 by the above new control input in 
Eq. 56 yields the following reduced system: 



Xr(t) = AorXrit) + Bor[-KXr(t) + r(t)] 
y{t) = CorXrit) + Dor[-KXr{t)+r{t)\ 

which can be re- written as: 



(57) 
(58) 
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Fig. 4 (Color online) 
Enhanced system step 
responses using pole 
placement; full-order system 
model {solid blue line), 
transformed reduced model 
{dashed black line), 
non-transfonned reduced 
model {dashed red line), and 
the controlled transformed 
reduced (dashed pink line) 
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Xr(t) = AorXrit) - BorKXr{t) + B„rr{t) -^ Xr{t) 
= [Aor - BorK]Xr(t) + Borr(t) 
yit) = CorXrit) - DorKXr(t) + Dorr{t) -^ yit) 
^[Cor - DorK]ir{t) + Dorr{t) 

The overall closed-loop model is then written as: 

'x{t)^AciXr(t) + Bcir{t) 
y{t)^CciXr{t) + Dcir{t) 



(59) 
(60) 



such that the closed-loop system matrix [Ac/] will provide the new desired eigen- 
values. As an example, consider the following non-scaled quantum system: 



A = 





142.857 



-0.385 
-18 



5 = 



0.077 




C = [0 1], 



D = [0] 



Using the transformation-based reduction technique, one obtains the reduced model 
{i,(f) = [-3.901]i,(f) + [-5.255]u{t),yr{t) = [-0.197]i,(f) + [-0.066]M(f)} 
with the eigenvalue of —3.901. Now, suppose that a new eigenvalue 1 = — 12 that 
will produce faster system dynamics is desired for this reduced model. This objec- 
tive is achieved by first setting the desired characteristic equation as A -I- 12 = 0. To 
determine the feedback control gain K, the characteristic equation is accordingly 
utilized by using Eqs. 57-60 which yields {(AI — Ad) = ^ XI — [Aq, — BorK] — 
0} after which the feedback control gain K is calculated to be —1.5413, and the 
closed-loop system now has the eigenvalue of —12. Simulating the reduced model 
using a sampling rate Ts — 0.005 second and a learning rate rj = 0.015 with the new 
eigenvalue for the same original system input (i.e., step input) has generated the 
response in Fig. 4. 
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6 Conclusions and Future Work 

A new method of intelligent control via neural estimation and LMl-based trans- 
formation for controlling time-independent quantum computing systems is imple- 
mented, and a simple state feedback control using pole placement was then applied 
on the reduced quantum computing model that achieved the required system re- 
sponse. Future work will investigate the implementation of the introduced hierar- 
chical control onto other quantum systems such as the non-linear, relativistic, and 
time-dependent quantum computing systems. 
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Optimal Guidance and Control for Space Robot 
Operation 



Takuro Kobayashi and Shinichi Tsuda 



Abstract This paper deals with a control of space robot for capturing moving tar- 
gets. It would be desirable to use the space robot to repair the failed satellite and 
to remove space debris since the work load to do these tasks by astronauts will be 
extremely heavy. Extensive studies have been done for the control of space robot. 
Unfortunately these studies have not incorporated the orbital motion which is essen- 
tial for space robot. Coplanar motion between space robot and target is discussed 
in this study. Suboptimal control, which uses piecewise optimized feedback gain by 
optimal tracking control method, is applied to chase the target. Also Hill's equa- 
tion was applied to the relative orbital equations of motions. Based on the above 
formulation dynamical simulation was conducted to demonstrate the validity of our 
approach. 

Keywords Hill's equation ■ Motion control • Optimal control ■ Space robot 



1 Introduction 

Recently space robots have been extensively used for the space activities like the 
International Space Station. Moreover Japanese government announced future space 
programs which will utilize the robot technology like for Moon exploration. 

A lot of studies have also been made for more advanced space robots like a robot 
satellite, in which the robot will be operated in an autonomous manner. These robots 
are expected to play an important role, like a space debris capture and retrieval. 
However these studies have not given any consideration about the effect of orbital 
motion, which generates the relative motion between the space robot and moving 
target. 
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Fig. 1 Model of space robot 
with end effector 



End Effector 
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Joint 2 




Joint 1 



Based on the above consideration the control of space robot, like a satellite robot, 
is discussed, in which its hand, an end effector of the space robot, tracks the moving 
target [1]. Firstly the kinematics of space robot is formulated using the generalized 
Jacobian [2]. And then the control method to correct the position error between 
the end effector and the target with feedback is defined. Dynamical equation was 
also derived to obtain the relation between joint variables and applied torque. In 
this development a linearized approximation, in which the centrifugal and Coriolis 
terms were neglected, was done by assuming the small deviation of joint variables 
and their velocity. The tracking control is formulated by applying optimal control 
theory, LQR [3]. Piecewise optimization for time-varying state-space equation is 
applied in this paper. This is a practical solution for suboptimal control and as shown 
in the simulation result it looks working well. Hill's equation was introduced to deal 
with the relative motion between the space robot and the target, which are assumed 
to be on a circular orbit without loss of generality. 



2 Model of Space Robot 



Figure 1 shows the model of space robot with an end effector. 

Two dimensional motion is assumed here, therefore, Oq gives the satellite attitude 
angle and two joint angles are defined to express the robot arm posture. 

The relation between the position of the end effector and joint variables is given 
by the following general equation: 



r = f(qM) 



(1) 



where r and Qm are the end effector position andjoint variable vectors, respectively. 
The velocity relation between end effector and the joint variables is as following: 



JmQm 



(2) 



Since the base of the space robot is not fixed, the satellite attitude is also changed 
by the arm operation. In this respect the generalized Jacobian was proposed to ana- 
lytically deal with this issue. Momentum and angular momentum conservations give 
us the relations between end effector andjoint variable velocity as below: 
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Fig. 2 Profile of Ar 



ds.r[mm 




r = J*qM 

J* = Jm - Jslg Im 

isQs + ImQivi = 



(3) 
(4) 
(5) 



where Jm: Jacobian matrix of the robot arm, Js: Jacobian matrix of the satellite, 
Im : robot arm inertia tensor and Ig : satellite inertia tensor. 
From (4) and (5) we obtain the following relationships: 



r-Ur 



Aqjvi 

Aqs = -ig VijviAqM) 



(6) 
(7) 



Thus when we define Ar, we can obtain Aqm and Aqg. 

Ar will be given at a small interval. This idea is shown in Fig. 2 as an example, 
in which 



Ar = 



Ar: 



Td 



Ar 



rt-Tdl 

f O.lf 
0.5 

|rt-rd| 



0<?<5 

5<f 
|rt-rd|<0.5 



in order to eliminate the error of the end effector (6) was modified as follows: 

AqM = r"^[Ar + X(rd-r)] 



(8) 



(9) 



(10) 



where k, r^ and r are feedback gain, goal position and current position, respectively. 



3 Dynamics of Space Robot 

The dynamics of the robot is generally expressed in the following form: 

M(q)q + h(q, q) = t 



(11) 



where M(q): inertia matrix, h(q, q): matrix derived from centrifugal and Corioli 
forces, and t: applied torque. The linearization was made in the vicinity of the 
current posture of the robot and the second term in left hand side of the above equa- 
tion was neglected by assuming both the small angle deviation and very slow joint 
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velocity. This will be shown in the simulation result. Thus we obtain a linearized 
dynamical equation as bellow: 



q = M(q)-lT 

The specific expression of this equation is given in Appendix 1 . 



(12) 



4 Optimal Tracking Control 



Linear optimal control, LQR, is applied to the control of the space robot to track the 
target. Generally state space representation for optimal tracking control is as fol- 
lows. The state equation and observation equation are given by the vector equations 
as below: 



X — Ax + Bu 

y = Cx 



(13) 



and when a desired trajectory qd(?) is defined in the time interval between to and 
tf, the tracking error is given by the following equation: 



e(0 = qd(0-q(0 



(14) 



where qd(0 = qd(«) = qd(« — 1) + [Aqs(f) AqM(0]'^ for the n-th step. And the 
performance index is defined in the quadratic formula: 



1 f'f 

/ = - / (e'^Qe + u^Ru) dt 

2 Jtn 



(15) 



where Q and R are a positive semi-definite symmetric and a positive definite sym- 
metric matrices, respectively. 

The solution x" of the above optimal control problem is resolved as the TPB VP 
(Two Point Boundary Value Problem) expressed by the following differential equa- 
tion: 



X — Ax -t- Bu 



X 

p. 


= 


A -BR-^B^" 
-C^QC -A^ 


X 

.P. 


+ 





5es the relation: 






u"(f) = -R-^B^] 


?{t) 







Qd 



Let us assume that p(r) is as below: 

p(f) = K(f)x(0 + pi(0 



(16) 



(17) 



(18) 



where K(?) is a symmetric and positive definite matrix with the same dimension as 
the vector x(r). Replacing p(f) by pi(f), we obtain the following state equation 
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X 

.Pi. 


= 


F -BR-^BT 
-G -FT 


X 

.Pi. 


+ 



CTq 


Qd 


(19) 


f^a-br-irTk 


(20) 


-G = 


= kbr-ibTj^ - ka - . 


\Tk 


-C 


tqc- 


k 


(21) 



Let us substitute the followings into (19) and (21): 

G = and K = 
Then we have the state equation and algebraic Riccati equation as below. 

Qd 
KBR-^B^K - KA - A^K - C^QC = 



X 




F 


-BR-^BT 


X 







Pi. 


— 


-G 


-FT 


.Pi. 


+ 


CTQ 



(22) 
(23) 



We apply a feedback gain K, which is given by the steady state solution, to our 
tracking control. 



5 Relative Motion by Orbital Motion 



The relative motion is described by Hill's equation with the reference coordinate 
system of the target. X axis is directed toward the target movement and Y axis is 
normal to the orbital plane. Z axis is reversely directed toward the earth center. 
Then we obtain the following equations: 



X = —gt 2(iL)z 

n 

y 

n 

z — 2gt Icox - 

n 



■ Cl>Z + Ol> X + Ax 



(24) 



-co Z' 



where gr is gravity acceleration, and A is external force, for example, by thrusters. 
As assumed in the previous section the target is on a circular orbit and external 
forces are not acting, then cd — O and a> — ^/gijrt- 



X + 2a>z — 
y + CD~y — 
z — 2(j)x — 3tti z — 



(25) 



The solution of these equations is shown in Appendix 2. In order to avoid the col- 
lision between the satellite and the target, the satellite robot arm will be approaching 
from the line of the target movement. And the coordinate system is converted to the 
space robot reference frame with X axis directed toward the reverse of the target 
movement and Y axis directed toward the reverse of the center of the earth. 
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Table 1 Satellite and robot 
characteristics 


Body 


Linkl 


Link 2 




Mass [kg] 1500 


30 


50 




Link length [m] 1.5 


1.5 


2.5 




Moment of inertia [kg m^ ] 1 000 


22.5 


26 




Initial angle [deg] -90 


90 


90 


Table 2 Target orbital 
properties 






Target 




Center of rotation: X direction [m] 




4.8 




Center of rotation: Y direction [m] 




1 




Rotational radius [m] 




1.5 




Altitude [km] 




1000 




Angular velocity [deg/sec] 




1 




Relative velocity: X direction [m/sec] 




-0.001 




Relative velocity: Y direction [m/sec] 




-0.0001 



6 Simulation Results 

The simulation, in which the end effector is tracking the target during 90 seconds, 
was conducted. 

We made the following assumptions for simulation study: 

( 1 ) The target is in reachable zone by the end effector of the robot during the period, 
and as noted above, 

(2) Only coplanar motions are allowed for the robot and target. 

Parameters used in the simulation are shown Tables 1 and 2. 
And the following parameters are assumed in the simulation: 

X = 0.05 



and 



e = 



1 

1 

LO 1. 



/? = 



1 0- 

1 

LO 1. 



Figure 3 illustrates the initial geometrical relation between the robot satellite and 
moving target. 

Figures 4 and 5 show the end effector has reached the target with an error less 
than 0.001 m at 45.6 second. The maximum velocity of the joints was smaller than 
0.05 rad/sec which is compatible with other space robot arm. 

Figure 6 illustrates the detailed relative distance between end effector and target 
from 45 second to 90 second. This chart shows good tracking performance. 
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Fig. 3 Initial geometry Y 

between target and satellite 
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Fig. 4 Positions of target and 
robot at 0, 30, 60, 90 seconds 



E, 

> 




Fig. 5 Relative distance 
between end effector and 
target 
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The histories of joint angles and their velocity, which were obtained from the 
kinematic equation (10), are given in Figs. 7 and 8. These are trajectories followed 
by tracking control. 
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Fig. 6 
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Fig. 7 History of joint angles 
by kinematics 
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Fig. 8 History of joint angle 
velocity by kinematics 
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Figures 9 and 10 illustrate the result of optimal tracking control, in which good 
agreement with the desired trajectories given by Figs. 9 and 10 is observed. And 
small angular velocities of joints are preserved. 
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Fig. 9 Joint angle history by 
optimal tracking control 
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Fig. 10 Joint angle velocities 
by optimal tracking control 
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7 Conclusions 



The space robot tracking control to capture a target was discussed. In general for 
orbiting targets like the space debris we have to consider two motions, rotation about 
their center of mass and orbital motion, at the same time. Especially targets like 
space debris and failed satellites are noncooperative for capturing them by space 
robot so that tracking control is inevitable. 

Kinematics and dynamics are formulated, including orbital motion which has not 
been discussed yet. And the optimal tracking control method was applied by using 
piecewise optimized feedback gains. 

Simulation result shows satisfactory performance of the control system by our 
approach. 
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Appendix 1: State Space Equation of Joint Variables for Space 
Robot 



d 

dt 
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A. 



10 



10 



1 

.0 OJ 



01 












i(q)-^ 


Mi2(q)-i 


Mi3(q)-i 











i(q)"^ 


M22(q)"' 


M23(q)-> 











i(q)-^ 


M32(q)-' 


M33(q)-i- 



to' 

■ r2. 



Appendix 2: Solutions of Hill's Equations 
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The Application of Genetic Algorithms 

in Designing Fuzzy Logic Controllers for Plastic 

Extruders 



Ismail Yusuf, Nur Iksan, and Nanna Suryana 
Herman 



Abstract This paper investigates tiie application of Genetic Algorithms (GA) in 
the design and implementation of Fuzzy Logic Controllers (PLC) for temperature 
control in an extruder. The importance of PLC is during the process of selecting 
the membership functions. What is best to determine the membership functions is 
the first question that has be addressed. It is important therefore to select accu- 
rate membership functions but these methods possess one common weakness where 
conventional PLC use membership functions generated by human operators. In this 
situation the membership function selection process is done by trial and error and 
it runs step by step which is too long to arrive at a solution to the problem. This 
research proposes a method that may help users to determine the membership func- 
tions of PLC using GA optimization for the fastest process in solving problems. The 
data collection is based on simulation results and the results refer to the maximum 
overshoot. Prom the results presented, the system arrives at better and more exact 
results and the value of overshoot is decreased from 1 .2800 for PLC without GA, to 
1.0011 for PGA. 

Keywords Temperature • Extruder ■ Fuzzy logic ■ Genetic algorithms • 
Membership function 



1 Introduction 

Automatic control has played an important role in the advance of engineering and 
science. Automatic control is essential in such industrial operations as controlling 
pressure, temperature, humidity, viscosity, and flow in the process industries. While 
modern control theory has been easy to practice [14], PLC have been rapidly gaining 
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Fig. 1 Fuzzy controller architecture 

popularity among practicing engineers. This increase of popularity can be attributed 
to the fact that fuzzy logic provides a powerful vehicle that allows engineers to 
incorporate human reasoning in the control algorithm. 

In our daily lives from manufacturing plant production lines, medical equipment, 
agriculture to consumer products such as washing machines and air-conditioners, 
FLC can be applied. A good example is the controller temperature set for plastic 
extruders by FLC [21]. When extruding certain materials, the temperatures along the 
extruder must be accurately controlled in accordance with properties of the extruder 
and the particular polymer. If the temperatures are not accurately controlled, the 
molten polymer will not be uniform and may decompose as a result of excessive 
temperatures. 

One of the problems associated with prior extruder control systems is the design 
of the barrel zone temperature controllers [11, 19]. Preferably, these controllers are 
designed with a high sensitivity to disturbance signals. However when a change in 
a temperature set point occurs, there is a danger of saturating the zone tempera- 
ture controllers as the magnitude of the temperature set point changes are generally 
greater than the magnitude of the disturbances. The sensitivity of the controller to 
disturbance signals must be reduced to prevent saturation of the controllers to set 
point changes [1, 11, 19]. It is therefore important to select accurate membership 
functions for temperature settings in extruder control systems. 

Taking the above explanation, we propose to use control systems based on FLC. 
The process of selecting membership functions is an important part of FLC [9]. 
With conventional FLC the membership function is generated by human operators 
manually. This has one common weakness since the selection process is done by 
trial and error and runs step by step which are too long in solving the problem [1] and 
interpretation mistakes may happen [18]. A new approach for optimum coding of 
fuzzy controllers is using GA to determine membership function especially designed 
for particular situations. We use GA to tune the membership function to terms of 
each fuzzy variable. 



2 Fuzzy Logic Controller 



Fuzzy logic process (fuzzy inferences) provides a formal methodology for repre- 
senting, manipulating, and implementing a human's heuristic knowledge about how 
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to control a system. The fuzzy controller is composed of the following four ele- 
ments: fuzzy rules, inference mechanism, fuzzification and defuzzification interface 
[15] as shown in Fig. 1. 

For the closed loop control system, the signal output is denoted by y{t), the input 
signal is denoted by u{t), and the reference input to the fuzzy controller is denoted 
by r{t). The actuating error signal is a difference both of input signal and feedback 
signal, it will be feeding into the controller to reduce error and bring the output of 
the system to a desired value [2]. 

We must have a basis for analyzing and comparing the performance of various 
control systems. Performance of various control system can be analyzed by concen- 
trating on maximum overshoot response as shown graphically in Fig. 2. 



3 Genetic Algorithms 



GA borrow ideas from and attempt to simulate Darwin's theory on natural selection 
and Mandel's work in genetic inheritance. The usual form of GA was described 
by Goldberg [6]. GA are stochastic search techniques based on the mechanism 
of natural selection and natural genetics. It differs from conventional search tech- 
niques. It starts with an initial set of random solutions called "population". Each 
individual in the population is called a "chromosome", representing a solution to 
the problem at hand. The chromosomes evolve through successive iterations, called 
generations. During each generation, the chromosomes are evaluated, using some 
measures of fitness [5]. To create the next generation, new chromosomes called 
offspring are formed by both crossover and mutation operations, and a new gener- 
ation is formed by selection and rejection [3, 12]. Fitter chromosomes have higher 
probabilities of being selected. After several generations, the algorithms converge to 
the best chromosome which hopefully represents the optimal solution of the prob- 
lem. 
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4 Materials and Methods 



Basically, GA has had a great measure of success in search and optimization prob- 
lem solving. In this research, GA are used to improve the performance of the fuzzy 
controller. Considering that the main attribute of the GA is its ability to solve the 
topological structure of an unknown system, then the problem for determining the 
fuzzy membership functions can also fall into this category. The concept of using 
GA for determining membership function was carried out early in a review [20] and 
was applied [9]. 

This research uses fuzzy genetic algorithms (FGA) for designing temperature 
control in an extruder. There are three possible zones in a thermoplastic screw: feed 
zone, melt zone and pressurizing zone [10, 11]. Each zone will be equipped with one 
or more thermocouples for temperature control. The pressurizing zone (also called 
metering or melt conveying) gives the plastic uniform pressure and flow characteris- 
tics. Research [21] was to design a system control based on PLC for controlling the 
temperature so the desired melting point in the "pressurizing zone" could be main- 
tained. The conceptual idea is to have an automatic and intelligent scheme to tune 
the fuzzy membership functions of the closed loop control for extruder machine 
using GA, as proposed in Fig. 3. 

A mathematical model of plastic polymer for a single screw extruder is given: 

(0.0035 ■ Qh) + (240 • r,„) + (0.000202 ■ r„) 



T{s) = 



6(5' + 240.000202) 



(1) 
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where: 

Qi, = heat input rate (kcal/sec) 

T„, = temperature of polymers (110°C) 

Tu — temperature of outlet air (40°C) 

T{s) — transfer function of temperature t(s) 

For obtaining final (tuned) membership function by using GA, some functional 
mapping of the system will be given. Parameters of the initial membership function 
are generated and coded as real numbers that are concatenation to make one long 
string to represent the whole parameter set of membership function. A fitness func- 
tion is used to evaluate the fitness value of each set of membership function. Then 
the reproduction, crossover and mutation operators are applied to obtain the optimal 
population (membership function). The final tuned value describes the membership 
function which is proposed. 

Having now learnt the procedures for designing a PLC, the practical realization of 
this system to determine the membership function in PLC is not an easy process. The 
dynamic variation fuzzy input of membership functions will be main the stumbling 
block to this design. 



4.1 Data Structures 

The most important structures in GA are those that represent genes and chromo- 
somes. Most researchers [1,4] have represented the chromosome string of genes as 
a binary code but in our case, the real numbers code will be used as real numbers 
code is a more natural representation than binary code [17]. 

This is research intended to determine membership function of the fuzzy parti- 
tion based on real numbers. Each gene corresponds to one linguistic variable whose 
definition is what the GA tries to evolve. In fuzzy systems, membership function 
was representing the value of a linguistic variable and for our simulation design, 
there are three kinds of linguistic variables: the variable error signal as input- 1 is 
parameter X and the rate of change in error as input-2 is parameter Y and the con- 
troller output is parameter Z, as shown in Fig. 4. For every variable there are five 
shapes of membership functions, three are triangular and two trapezoid. If mem- 
bership function has triangular form, then it can be described by three parameters. 
A fixed number of real numbers is used to define each of the three parameters which 
completely defines the specific triangular membership function. If it is trapezoidal, 
it requires four parameters. 



4.2 Fitness Function 

In the evolution of nature, the highest valuable individual fitness will survive 
whereas the low valuable individual will die. The fitness function is the basis of 
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Fig. 4 Structure of chromosome 

the survival of the fittest premise of genetic algorithms. It is responsible for evalu- 
ating the parameter sets, and choosing which parameter sets are suitable. The most 
difficult part of the fitness function is to design the function for producing param- 
eters that are reliable and effective. The fuzzy controller operates in a closed-loop 
specification, which can be analyzed by the maximum overshoot. The maximum 
overshoot {Mp) is the maximum peak value of the response curve measured from 
the unity and the amount of the maximum (percent) overshoot directly indicates the 
relative stability of the system [1, 2, 14, 16]. 

The fitness function measures how close each individual in the population meets 
the given specification at a given instant of time. Thus for a given individual in 
the population, the maximum overshoot is computed respectively by performing a 
simulation of the closed loop control system with the candidate controller (FGA) 
and model of the plant. 



4.3 Genetic Parameters 



The stumbling block for implementing GA is to set the values for the various pa- 
rameters, such as population size, probability of crossover rate, and probability of 
mutation rate. These parameters typically interact with one another nonlinearly, so 
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they cannot be optimized for all situations. This research is set to population sizes 
of 5, 10 and 100 for comparing, while the probability of crossover rate is 0.1, 0.7 
and 0.9 where the probability of mutation rate is 0.001, 0.05 and 0.1. 



4.4 Termination Condition 

GA will typically run forever until an appropriate termination condition is reached. 
Termination criteria in the GA literature are somewhat arbitrary because the statis- 
tical nature of the search prohibits knowing when the population will evolve to a 
given fitness value. Typically, GA searches are terminated after [3, 18]: 

1 . A maximum number of generations or individual population members have been 
evaluated, or 

2. The diversity of the string values in the population has been significantly reduced, 
or 

3. Goal fitness (such as 1.0) has been achieved. 

The termination condition for this research is the one that defines the maximum 
number of generations to be produced. When the generation number is completed 
by the GA, the new populations generating process is finished, and the best solution 
is the one among individuals that most adapted to the evaluation function. 



5 Result and Analysis 

As mentioned previously, the most important aspect in implementing GA for im- 
provement of performance is how to set values of the various parameters, such as 
population size, the probability of crossover and the mutation rate. These parameters 
typically interact with each other. 

• P' experiment: combination of population size: (5: 0.7: 0.001), (10: 0.7: 0.001) 
and (100: 0.7: 0.001). 

• 2"'' experiment: combination of mutation rate: (10: 0.7: 0.001), (10: 0.7: 0.005) 
and (10: 0.7: 0.05). 

• 3'^'* experiment: combination of crossover rate: (10: 0.1: 0.001), (10: 0.7: 0.001) 
and (10: 0.9: 0.001). 

• 4* experiment: combination of crossover rate: (10: 0.1: 0.05), (10: 0.7: 0.05) and 
(10: 0.9: 0.05). 

Referring to previous available research [7, 8, 13, 20], it seems (10: 0.7: 0.001) to 
be the best combination rate and is chosen for the tests although it does not give 
optimum results in the further tests. 

In the first experiment we compare the convergence rates of populations: 5, 10 
and 100 individuals while the probability of crossover rate is 0.7 and the probability 



32 



I. Yusuf et al. 



(Population Size: 0.7: 0,0011 



00930 - 



£ 00 



0,0830 



0,0780 




11 16 

Generation 

a 



(10 : 0.7 : Mutation Rate) 



y, 0.0930 
W 

E 00880 



0.0830 



007S0 



-•-(10: 0,7: 0.001) 

-^(10: 07: 0.005) 

(10: 0.7: 0.051 



jii t ri-^ ■ ' ■ 



*t 



t*j**<ntt-yn 



11 16 

Generation 
b 



Fig. 5 Comparison of convergent: (a) Population size, (b) mutation rate 



of mutation rate is 0.001. All the data can be shown in the graph in Fig. 5a. These 
results show that the performance of GA for first generation with a population size of 
5 and 10 are the same i.e. 0.0798. Then this result will increase in the next generation 
until final generation. In the training data, we found the fitness 0.0823 at the 25* 
generation when the population size is 5 and 0.0828 at the 25'^ generation when 
the population size is 10. For the best result, the fitness is 0.0999 at 25* generation 
when the population size is 100. But the consume time varies significantly, from 
370 minutes when the population size is 10 and increases to 1205 minutes when 
the population size is 100. These results show that the performance of GA is very 
sensitive to the population size. 

In the second experiment, we make a comparison of the convergence of mutation: 
0.001, 0.005 and 0.05 while the probability of crossover rate is 0.7 and the popu- 
lation size sets to 10 individuals. All the data can be shown in the Fig. 5b. From 
Fig. 5b, we can see the effect of a probability mutation rate on the fitness value. If 
the value of mutation rate is high then the fitness value will get better. The highest 
fitness value is 0.0925 for a probability mutation rate of 0.05, secondly it is 0.0833 
(probability mutation rate of 0.005) and the lowest value is 0.0828 (probability mu- 
tation rate of 0.001). 

If the probability mutation rate is high then the time consumed will increase. 
From 370 minutes (when the probability of mutation rate is 0.001) it increases to 384 
minutes (when the probability of mutation rate is 0.005) and 467 minutes (when the 
probability of mutation rate is 0.05). This situation shows that GA need a longer time 
if probability of mutation rate is higher. In this case, the combination of parameters 
(10: 0.7: 0.05) shows the best performance of GA while the combination parameter 
(10: 0.7: 0.001) shows the lowest performance of GA. This is different with our 
hypothesis. 

Result of the third experiment can be show in the following graph in Fig. 6a. Fig- 
ure 6a shows a comparison of convergence of crossover rate: 0. 1 , 0.7 and 0.9 while 
the probability of mutation is 0.001 and the population size sets to 10 individuals. In 
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Fig. 6 Comparison of convergent while mutation rate set to: (a) 0.001, and (b) 0.05 



this experiment, the combination parameter (10: 0.9: 0.001) shows the best perfor- 
mance of GA, while the combination parameter (10: 0.1: 0.001) shows the lowest 
performance of GA. 

We can now see the effect of probability crossover rate on the fitness value. If 
the value of crossover rate is high then the fitness value will get better. The high- 
est fitness value is 0.0867 for a probability crossover rate of 0.9, secondly it is 
0.0828 (probability crossover rate of 0.7) and the lowest value is 0.0826 (proba- 
bility crossover rate of 0.1). If the probability crossover rate is high then the time 
consumed will increase. From 316 minutes (when the probability of crossover rate 
is 0.1) increases to 370 minutes (when the probability of crossover rate is 0.7) and 
425 minutes (when the probability of crossover rate is 0.9). This situation shows 
that GA need a longer time if probability of mutation rate is higher. 

In the fourth experiment, we make a comparison of the convergence rates of 
crossover rate: 0. 1 , 0.7 and 0.9 while the probability of mutation is 0.05 and the 
population size sets to 10 individuals, as shown in Fig. 6b. In this experiment, the 
highest fitness value is 0.0929 while combination of parameters are set to (10: 0.9: 
0.05), the second fitness value is 0.0925 while combination of parameters are set to 
(10: 0.7: 0.05), and the lowest fitness value is 0.0921 while combination of parame- 
ters are set to (10: 0.1: 0.05). 

The comparison between Fig. 6a and 6b shows the fitness value only slightly 
different for any value of crossover rate if the probability mutation rate is set to 0.05 
(in Fig. 6b). Figure 6a shows it has significant fitness value if crossover rate sets 
to 0.9 compared to 0.1 and 0.7. This situation shows that the values of probability 
crossover rate and mutation rate interact, both will affect each other. 

From the results presented in this experiment, the system which we have devel- 
oped is very helpful to determine membership function for the fastest processing in 
solving problems. Figure 7 (a, b) shows the membership function existing (without 
GA) and will be used in initial population in the first chromosome, first population 
and first generation. 
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Compare with Fig. 8 (a, b), where GA were applied into the fuzzy system to 
determine the membership function (proposed membership function). 

During the execution of FGA, the fitness result has been recorded as data. The 
membership function evolves after evolution is complete and will be tested using the 
data for comparing the PLC with and without GA. Generally speaking, after mem- 
bership function has been tuned by using GA then it will improve the performance 
of a PLC. We can compare the PLC with and without GA thus the visualization 
as shown in Pig. 9. After programming is executed then membership function is 
regulated automatically. A better and more exact result is obtained: the value of 
overshoot is 1 .2800 for PLC without GA (shown in Pig. 9a) and decreases to 1 .00 1 1 
for PGA (shown in Pig. 9b). It is clear the GA are very promising to improve the 
performance of a PLC and for getting more accuracy in order to achieve optimum 
results. 
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Fig. 9 Response system for membership function: (a) Without GA, (b) with GA/FGA 

6 Discussion 



GA has been successfully applied to solve optimization problems. In this research, 
GA are implemented for determining the membership function of FLC in a plastic 
extruder to control the temperature. By designing compact data structures for genes 
and chromosomes and an accurate fitness evaluation function, GA have been imple- 
mented to effectively find more accurate membership functions for the fuzzy system. 
The data structures adopted are compact and thus very conveniently manipulated by 
genetic operators. 

Population size is a significant factor to improve the performance of GA. Gen- 
erally speaking, the larger population size will be better for performance of GA; a 
larger population size has more diverse chromosomes and thus will contain a large 
number of good chromosomes. This is helpful to achieve a better solution. But it also 
increases the time required for the population to converge and it causes longer pro- 
cessing time. The population size specifies how many organisms should be stored 
in memory at any given time. 

The selection operation is the probability of the chromosome (offspring) for sur- 
viving and will be used in the further chromosome calculations (crossover and mu- 
tation) for the next generation. Mutation serves to create random diversity in the 
population, while crossover serves as an accelerator that promotes emergent behav- 
ior of components. Increasing the mutation probability will increase areas of random 
search, but it also makes for a loss of the genetic material. Increasing the crossover 
probability will increases the recombination of chromosomes, but it also disrupts 
good chromosomes. 

This research found several matters which are important factors that influence 
performance of GA. There are three basic issues: 

1 . Selection procedure for choosing suitable candidates for mating. 

2. The value of a for scaling factor in arithmetic crossover method. 
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3. The value of b for parameter degree in dynamic (or non-uniform) mutation 
method. 



7 Conclusion and Future Research 

The population size was a significant factor to improve the performance of GA. The 
larger population size will be better for performance of GA, but longer in processing 
time. GA need a longer time if probability of crossover and mutation is higher. The 
interaction between probability of crossover and mutation is significant; both of 
them will affect each other. In this research the following significant features have 
been identified: 

• Combination parameter (10: 0.9: 0.05) will give the best value for solving our 
problem. The value of fitness is 0.0929 (maximum overshoot = 1 .0770) and the 
processing time is 580 minutes. 

• Combination parameter (100: 0.7: 0.001), will give the best fitness value. The 
value of fitness is 0.0999 (maximum overshoot — 1.0011) and the processing 
time is 1205 minutes. 

The performance of GA can be further improved by using different combinations of 
selection strategies, crossover and mutation methods and other genetic parameters 
such as population size, probability of crossover and mutation rate. 
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Automatic Weight Selection and Fixed-Structure 
Cascade Controller for a Quadratic Boost 
Converter 

Somyot Kaitwanidvilai and Pitsanu Srithongchai 



Abstract In this paper, a new technique for designing a robust cascade controller 
for a quadratic boost converter is proposed. A single performance index in H infin- 
ity loop shaping control called stability margin is adopted as the objective function 
in the proposed optimization control problem; GA is used to solve this problem to 
evaluate the optimal controller. Necessary conditions of cascade control scheme are 
adopted as the constraints in the optimization problem. In addition, pre-compensator 
weight, which is normally difficult to be selected, is simultaneously determined with 
the controller. Comparative study with the conventional H infinity loop shaping is 
presented. Finally, simulation results verify the effectiveness of the proposed tech- 
nique. 

Keywords H infinity loop shaping control ■ Genetic algorithm • Quadratic boost 
converter ■ Fixed-structure robust control 



1 Introduction 

Several advantages of quadratic DC-DC converter such as reduction of the reso- 
nance mode in DC-DC converter, simple circuit, etc. have been presented in the 
previous research works [1-4]. The design of robust controller for this converter is 
needed to enhance both the performance and the robustness of the controlled sys- 
tem. Although the standard technique such as H infinity optimal control can provide 
a feasible way to design the robust controller; however, the resulting controller in 
this approach is normally complicated with high order, making it difficult to im- 
plement in practice. In addition, weight selection in this technique, which is not an 
easy task, is normally carried out by trial and error method and needs the expert 
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knowledge to find the appropriate weights. To overcome this problem, we propose 
an algorithm, a robust cascade controller designed by GA, to design a robust con- 
troller for a quadratic DC-DC boost converter. In the proposed technique, inverse 
of infinity norm from disturbances to states is formulated as the fitness function 
in GA. The advantages of simple structure, controller structure selectable, and ro- 
bustness are achieved by the proposed technique. In addition, performance weight, 
which is normally difficult to be obtained, is simultaneously determined by GA. 
This reduces the difficulty of weight selection in the conventional robust loop shap- 
ing design. 

The remainder of this paper is shown as follows. Section 2 illustrates the con- 
verter model and the conventional H infinity loop shaping technique. Section 3 
describes the proposed technique. GA is also briefly described in this section. 
Simulations and results are shown in Sect. 4. Finally, Sect. 5 concludes the pa- 
per. 



2 Converter Model and Conventional Loop Shaping Technique 
2.1 Converter Model 

Figure 1 shows the typical circuit of a quadratic boost converter. The dynamic model 
of current and voltage loops in the cascade control scheme of this circuit can be 
expressed as [1, 2, 5]: 

_ is{s) _ a-js'^ + ais'^ + a\_s + ai^ 

d{s) s^ + b^s^ + b2S^ + bis + bo 

v(s) mi,s + m2S + mis + mo 

Gv — ~ — ^ ^ (2) 

i^(s) ass^ + ajs^ + ais + ao 
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Fig. 1 A quadratic boost 
converter 
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D, d are nominal duty cycle and duty cycle, respectively; Li, L2, Cu C2, R are the 
component values of the converter shown in Fig. 1; £ is the nominal input voltage; 
Gil and G^ are the dynamic models of current and voltage loops; ig —in + ill- 



2.2 The Conventional H^o Loop Shaping Control 

H infinity loop shaping control was first introduced by McFarlane [6]. In this design, 
desired open loop shape in frequency domain is specified by shaping the open loop 
of the system, G, with the weighting functions, pre-compensator {W\) and post- 
compensator (W2). The shaped plant can be written as: 



Gs^WiGW2 
Ga 



(3) 

: (N, + AnsKM, + Ams)'^ (4) 

where Ans and Ams are the uncertainty transfer functions in the nominator 
and denominator factors, respectively. Ga is the shaped plant with uncertainty. 
II A^Ti, A^illoo < s, where e is the stability margin. The design steps of H infinity 
loop shaping can be briefly described as follows: 

Step 1 Specify the pre- and post-compensator weights for achieving the desired 
open loop shape. 

Step 2 Find the optimal stability margin (Sopt) by solving the following equation. 

/~ 
K 



Yopt^s ,^ iiif 



stabK 



{i-GsK)-'m: 



(5) 



If {Sgp,) is too low, then go to Step 1 to select the new weights. 

Step 3 Select the stability margin (e < Egpt) and then synthesize the controller, 
^co, by solving the following inequality. 
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Fig. 2 Cascade control 
scheme for a quadratic boost 
converter Vr 



I current loop 

Gpm — 1»0 — ^ Gain | — ^\ Gil 



[l^ 



Gv 



Step 4 Final controller (K) is 



K = WiKocW2 



(7) 



3 Fixed-Structure Robust Loop Shaping Control 



Although the robust loop shaping technique is an efficient technique to design a ro- 
bust controller; however, the final controller designed by this approach is usually 
high order and complicated. To overcome this problem, we propose a GA based 
fixed-structure robust loop shaping control to design a fixed-structure robust con- 
troller. The proposed technique can be described as follows. 

The structure of the proposed controller is shown in Fig. 2. 

In this paper, we selected PI controller as the outer loop controller and P con- 
troller as the inner loop controller. If Ginnerioop is the transfer function of the closed 
loop system of the current loop, thus, G^ — W\GinnerloopGv and the stability margin 
in (6) can be written as: 



Wr'Gpn 



{l-GsW;^Gpn) '[/ GJ 



(8) 



Consequently, our design objective is to find the optimal gains of the P controller 
of current loop, the PI controller (Gpji), and the weight Wi such that the stability 
margin in (8) is maximized. To achieve the cascade controller, a constraint, which 
is "the bandwidth of inner loop must be much higher than that of the outer loop" is 
added to the optimization problem. The following steps are the proposed design. 

Step 1 Specify the structures of weight and controller. Select the post- 
compensator weight as /. 

Step 2 The structure of the voltage loop controller is Gpn (p), thus, based on (7), 



Koc^W(x)-'Gpn(p) 



(9) 



By substituting (9) into (6), the infinity norm of transfer function from distur- 
bances to states, subjected to be minimized, can be written as: 
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where Gs — WlGmnerloopGy. 
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Fig. 3 Fitness versus 
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Subject to BW (inner loop) > 5BW (outer loop). BW is denoted as bandwidth. 

Step 3 Use GA to find the optimal parameter, p*, gain, and x*. The foUowings 
briefly describe the GA. 

In the proposed technique, GA is adopted in the control synthesis. This algorithm 
applies the concept of chromosomes and the genetic operations of crossover, mu- 
tation and reproduction. At each step, called generation, fitness value of each chro- 
mosome in population is evaluated by using fitness function. Chromosome, which 
has the maximum fitness value, is kept as a solution in the current generation. The 
new population of the next generation is obtained by performing the genetic oper- 
ators such as crossover, mutation, and reproduction. In this paper, a roulette wheel 
method is used for chromosome selection. In this method, chromosome with high 
fitness value has high chance to be selected. Operation type selection, mutation, re- 
production, or crossover depends on the pre-specified operation's probability. Nor- 
mally, chromosome in genetic population is coded as binary number. However, for 
the real number problem, decoding binary number to floating number is applied. 



4 Simulation Results 

In our study, the converter parameters are given as follows: Ci — 22 pF, C2 — 
100 nF, Li = 90 nH, L2 = 382 ^H, load R = 100 ohms. GA is adopted to find the 
solution of above optimization problem. Weight parameters, gain in PI controller, 
and gain in P controller, are set as the chromosome in GA. Constraints of time do- 
main specifications, i.e. settling time <0.05 sec, overshoot <0.5% are also adopted 
in this optimization problem. When running GA for 49 generations, an optimal so- 
lution is obtained. 

Resulting weight and controller are shown in Table 1. Conventional H infinity 
control with the same weight and inner loop current controller gain is adopted to 
design the controller for comparison purpose. The full order H infinity controller is 
designed as (11). 



Hoo = ^ (11) 



where 

(0.5665i + 18.05) 
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Table 1 Resulting controllers and their stability margins 
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Proposed controller 



Conventional Hqo loop shaping 



Weight 


(0.5665.9 + 18.05)/i 


(0.5665.S + 18.05)/i 


Controller (outer loop) 


(0.4418.s+53.97)/i 


H<^in(ll) 

(9th order controller) 


Controller (inner loop) 


2.74 


2.74 


Stability margin 


0.70 


0.745 



Fig. 4 Step response of the 
proposed controller and H^o 
controller 
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5^ + 1.084 X 10V + 1.068 X 10^°i^ + 5.491 x 10'V+ 4.799 x lO'V 
+ 6.984 X lO^^i^"* + 5.482 X 10-V + 1.138x 10^^.^ + 3.113 x 10^° 



: 6037i^ + 6.499 X 10V + 1.223 x lO".?^ + 3.214 x 10^^/ 
+ 3.013 X 10^^^^ + 3.968 x lO^^s- + 8.896 x lO^*^ + 3.166 x 10 
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As seen in this table, the stability inargin of the proposed controller is almost the 
same as the conventional full order H infinity controller; however, the order of our 
controller is much lower than that of the conventional controller. This makes it easy 
to be implemented in practice. Since analog controller is normally used to design 
the converter controller, our technique is more feasible than the conventional robust 
control. 

Time domain responses of both controllers are shown in Figs. 3 and 4. As seen 
in these figures, our proposed technique gains better response in terms of fast rise 
time and fast settling time. 



5 Conclusions 



In this paper, the design of high-performance and robust controller for a quadratic 
DC-DC converter using Genetic Algorithm has been proposed. Results show that 
the order of the proposed controller is much lower than that of the conventional 
robust loop shaping controller. In addition, by the proposed technique, performance 
weight which is not easy to be specified, can be simultaneously evaluated with the 
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controller. The tracking performance specifications can be adopted as the constraint 
in the proposed optimization problem. 
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Availability Studies and Solutions for Wheeled 
Mobile Robots 

Adrian Korodi and Toma L. Dragomir 



Abstract The need of an increased availability in the field of robotics is essential. 
The subjects of this work are the wheeled mobile robots, and their key feature is 
the movement. The movement has to be realized as safely as possible. The safety 
implies faultless behavior, which means achieving the specification limits in any 
situation. A critical part in the wheeled mobile robots movement is the localization 
module. The first objective of the paper is to provide an availability analysis for 
wheeled mobile robots regarding the localization module. The first analysis focuses 
on a distance sensor used to detect static or dynamic obstacles from the environment 
and the second analysis is related to the entire localization module, the main part 
of the faults coming from odometry errors. The second objective of the paper is to 
realize a synthesis of redundant procedures meant to increase the value of the overall 
availability of the mobile robots. 

Keywords Availability • Fault tolerance ■ Markov models ■ Odometry errors ■ 
Wheeled mobile robot 



1 Introduction 

A high reliability and availability level for robotic systems is an increasing demand, 
especially for wheeled mobile robots which are moving in completely unknown or 
partially known environments. The mobile robots are repairable systems, and there- 
fore the availability is essential. The researches focused on reliability and availabil- 
ity analysis are in a small number [1, 3, 4, 7, 9] and many of them are incomplete. 
They are pointing out a reduced level of availability and a need of redundant struc- 
tures in order to increase it. 
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One critical and essential point for the mobile robots which are moving in par- 
tially known environments is the localization module. If it fails, the entire system 
fails, not being able to determine its own position, to avoid static obstacles, or not 
being able to detect the approaching mobile objects. The failures in the localization 
module are representing 25% of the total number of failures. 

In consequence an availability analysis method based on Markov models, and 
two availability analyses for wheeled mobile robots will be provided in this paper. 
Both availability studies are focused on the localization module. The first study de- 
termines the availability for an average distance sensor (present in every wheeled 
mobile robot), and it points out the availability after a fault tolerant controller is im- 
plemented [5] in order to tolerate certain types of faults of the distance sensor. The 
second study focused on the entire localization module and it provides an availabil- 
ity study for a wheeled mobile robot which is moving in partially known environ- 
ments. Also, being known that the majority of the localization failures are caused by 
odometry errors, the study determines the availability of the wheeled mobile robot 
after a correction module based on image processing is implemented [6], showing 
the results in a comparative manner. 



2 First Availability Analysis 

The current paragraph realizes an availability study over a usual distance sensor used 
in the localization module by the wheeled mobile robots. The mean time between 
failures (MTBF) is difficult to estimate because of the various types of sensors which 
may be used. Regarding the various received information regarding the MTBF for 
an average distance sensor, the following time values will be considered: 6 months, 
respectively 5 months. 

In the first situation the MTBF will be 180 days and therefore the failure rate 
will be Xi — 0.000231 hours" ^. Regarding the fact that the paper wants to point 
out also how the availability can be increased, types of failures are defined and the 
corresponding failure rates are analyzed. The types of possible faults are [5]: 

• a sudden fault is defined as the variation of the amplitude of the signal provided 
by the sensor (e) greater than a prescribed value {pa), over consecutive sampling 
periods: 

e[t] - e[t - 1] > Pa (1) 

• an evolutive fault is characterized by a temporized failure of the sensor, caused by 
certain wear out, pointed out through an amplified or attenuated signal provided 
by the sensor. 

A permanent fault is defined as a sudden fault maintained over np sampling pe- 
riods or as an evolutive fault which exceeds the acceptable limits. 

Examining the general MTBF and following a data analysis received from var- 
ious sensor producers, the author appreciates that the failure rates for evolutive. 
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sudden and permanent faults are k^ — 0.000124 hours~\ A;, — 0.000077 hours"', 
respectively kp = 0.000086 hours"' . 

The second situation corresponds to a 150 days MTBF, resulting the follow- 
ing general failure rate X, — 0.000277 hours"'. On the same bases that for the 
first situation, the following failure rates have resulted: kg — 0.000123 hours"' 
for evolutive faults, Xb — 0.000079 hours"' for sudden faults, respectively Xp — 
0.000125 hours"' for permanent faults. 

Regarding the repair rate for the sensors, following the affirmations made by 
big sensor producers (Schneider, Massa, etc.), the sensors must be almost always 
replaced after a failure, the repairing being impossible. Therefore, after a fail- 
ure at the distance sensor, regardless of its type (ultrasonic or infrared), a mini- 
mum 2 weeks mean downtime (MDT) is estimated. This downtime includes the 
buying and the shipping of the new sensor and also the faulty sensor replac- 
ing procedure. So, the considered repair rate for the availability study will be 
/i = 0.0029 hours"'. 

In order to realize an availability study, the Markov modeling will be used. The 
system's states are: 

• state © - the state of system success, the system is not affected by any type of 
fault at the distance sensor; 

• state ® - the state in which the distance sensor suffers an evolutive fault; 

• state (2) - the state in which the distance sensor suffers a sudden fault; 

• state Q - the state in which the distance sensor is affected by a permanent fault. 

The corresponding model is illustrated in Fig. 1 : 



Fig. 1 The established model 
for the availability analysis 




The transitory processes associated with the probabilities of the system to be 
in one of the four states of the model presented in Fig. 1 can be described by the 
following equations: 



Poit + Af) = Po(0(l - >-p^t - XhAt - AgAr) 

-I- Pi{t)ixAt + P2(t)iJ.At + P3(t)ixAt 



50 A. Korodi and T.L. Dragomir 

Pi(t + At) = PoCO^.Af + A (0(1 - mAO (2) 

P2(t + At) = Po(t)XbAt + P2it)(l - iiAt) 
P3it + At) = Po{t)\pAt + P3(t){l - iiAt) 

Realizing At — > 0, the state equations are obtained. These are helping in the 
study of the transition dynamics and availabihty evaluation: 

[ Poit) = -(Xe + Afe + Xp) ■ Po(t) + M ■ (Pi(0 + P2(0 + ^3(0) 

Piit) ^ X, ■ Po(t) - fx ■ Pi(t) ^^^ 

hit) = Xb ■ Po(t) - /X ■ P2(t) 
[P3(t)^Xp-Po{t)-fl-P3(t) 



3 First Availability Improvement 

The paragraph wants to emphasize one possibility to improve the availability hav- 
ing the distance sensor and its possible faults described in Sect. 2. A possibility to 
increase the availability level in the current situation is to use a fault tolerant con- 
troller like the one it is proposed in [5]. The purpose is not to detail the referred 
paper and therefore the following brief description is provided: the evolutive and 
sudden faults can be tolerated using a fault tolerant controller. The fault tolerant 
controller developed in [5] is composed by: a basic interpolative controller which 
uses a three-dimensional table, a reference-based redundancy to detect the faults and 
a correction module in order to provide the correct input for the basic interpolative 
controller if the distance sensor suffers an evolutive fault. 

The paragraph will do a comparative availability analysis, for the situation pre- 
sented in the previous paragraph and the situation in which a fault tolerant controller 
is used to tolerate evolutive and sudden faults. There will be two case studies regard- 
ing the distance sensor data regarding from Sect. 2. 

Figure 3 illustrates in a comparative manner the availability evolutions. Ainitiai 
represents the initial systems availability (without tolerating sudden and evolutive 
faults), and Agnal; represents the availability of the improved system. 

In order to calculate Ainitiai, state @ was considered the only success state (be- 
cause the evolutive and sudden faults are not tolerated by the initial system). When 
calculating Aflnai availability, the success states of the system were @, (i) and @, 
because the system tolerates evolutive and sudden faults of the distance sensor. 

The Simulink scheme necessary for the study, corresponding to the formula (3), 
is illustrated in Fig. 2. 

As it can be seen in Fig. 3, the availability is improving by 6.5% when the fault 
tolerant controller is used. 

For the second situation (MTBF =150 days), the concept is the same and after 
the simulations, the following evolutions (Fig. 4) are obtained for the availabilities 

Ainitiai and Afinal- 

As it can be seen in the figure, the availability is improving with 6-6.5% when 
the fault tolerant controller is used. 
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Fig. 2 The Simulink scheme associated with the availability analysis 
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4 Second Availability Analysis 

The following two paragraphs are realizing an availability and reliability study re- 
garding the influence of localization error correction during the movement of a 
wheeled mobile robot. 

The literature provides an extremely small number of availability studies for 
wheeled mobile robots which are moving in partially known environments. The ones 
useful for the research within this paragraph were only [1, 4, 9]. In the mentioned 
references, various data was presented regarding the reliability and availability of 
the wheeled mobile robots which are moving in partially known environments. The 
specific data in order to realize an availability study for different types of robots can 
be obtained only experimentally, after a long enough monitoring process. 

As being mentioned in [1, 4, 9], the MTBF indicator for wheeled mobile robots 
which are moving in partially known environments is: 7 hours (from the research 
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Fig. 4 The availability 
evolutions for the second 
situation study (MTBF = 150 
days) 
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[9]) and 19.5 hours (from the research [3]). From the study realized in [1] a MTBF 
of 17.5 hours was determined (the robot is functioning over a period of 140 hours 
and within this period 8 faults were occurring). 

To be able to realize the current study, it was necessary to determine the percent- 
age of the total failures which are caused by an erroneous localization. From [9] 
and [1], the following conclusion can be extracted: approximately 25% of the total 
failures are localization failures which can be corrected using the method presented 
in the next paragraph. 

The information until this point allows accomplishing a reliability study. But, 
the mobile robot is a reparable system and therefore a complete study must present 
an availability analysis. To realize an availability study, the MDT indicator must be 
known. In [4], the MDT indicator is provided, being MDT = 60.5 hours. The studies 
from [3, 4] are covering only the analysis of the inherent availability, although the 
operational availability is the one that is relevant. From the inherent availability 
study the value of the MTTR indicator is extracted, MTTR =1.3 hours. 

The reliability and availability study is based on Markov models. The number 
of states considered for a minimal analysis in this case is two: state ©, the state of 
success, and state ®, the failure state. The model from Fig. 5 is considered: 

Fig. 5 The established model 
for the reliability and the 
availability study 



The transitory processes associated with the probabilities of being in one of the 
two states represented in Fig. 5 can be described by the following equations, which 
are taking into account the corresponding failure and repair rates: 

Poit + At) = Po(0(l - ^01 Af) + Pi(f)/LtioAf 

(4) 
Pi(f + At) = Poit)XoiAt + Pi(f)(l - /xioAf) 

Realizing At -^ 0, the state equations are obtained. These are helping in the study 
of the transition dynamics, reliability and availability evaluation: 

hit) ^^01 Poit) -t^ioPi(t) 




(5) 
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Fig. 6 Modifying the 
orientation of the wheeled 
mobile robot 
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5 Second Availability Improvement 



In the previous paragraph, the percentage of locahzation failures from the total num- 
ber of failures was found out to be significant. In the case of the wheeled mobile 
robots, a big number of localization failures are caused by odometry errors. The 
systematic odometry error corrections are studied in [2, 8], realizing a priori cali- 
bration. There is a necessity to correct also the nonsystematic odometry errors and 
obviously, the correction has to be realized online. An online correction module 
based on image processing is proposed in [6]. This method is suitable for wheeled 
mobile robots that are moving in dynamic environments. Using a camera placed on 
the mobile robot, images are taken over from visual reference (target) points placed 
priori in the environment. If a fault that can be caused by odometry errors is detected, 
the orientation of the mobile robot towards the next reference point is corrected as 
in Fig. 6 (identifying the visual reference point through image processing), and the 
robot will be able to accomplish its task. 

Implementing the module described in [6], the localization for the wheeled mo- 
bile robots which are moving in partially known environments is improved and the 
overall availability and reliability of the system increases. A comparative availability 
and reliability study is realized in order to show the efficiency of using a redundant 
method in the localization module. 

Four experiments are realized: one reliability and inherent availability study and 
two operational availability studies. 

For the experiments, the transitions provided by the formulas (5) and (6) are 
implemented in Simulink, like shown in Fig. 7. 

The first experiment realizes a minimal wheeled mobile robot reliability analysis 
using the data from [3], where the MTTF is 19.5 hours (in the [9] study the situation 
is worst having a MTTF = 7 hours). Off course that in order to realize a reliability 
analysis, the repair rate from (5) is considered zero. Practically, the equation which 
guides the study is: 



Po(t)^-kQiPo(t) 



(6) 



Figure 8 shows the comparative reliability evolutions for the two situations: the 
basic system and the system where the redundant orientation correction method 
based on image processing is implemented. 
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Fig. 7 The Simulink scheme 
used to analyze the 
availabihty and the rehability 
of the system 
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The notations are the followings: 



• the analyze referring to the initial system - Sinitiai 

• the analyze referring to the system foreseen with redundant module ■ 



^ final 



Like it was mentioned before, the MTTF is 19.5 hours, and the failure rates have the 



-1 



following values: Aqi = 0.05128 hours for Sinitiai, and Aoi = 0.03846 hours for 

Jfinal- 

The following two experiments have as objective, the comparative study of the 
operational availability of the wheeled mobile robot for the two implemented situa- 
tions, Sinitiai and Sfinal- 

The first experiment shows an availability analysis where the MTBF is 19.5 hours 
for Sinitiai, and consequently its failure rate is Aqi = 0.05128 hours"^ In this 
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Fig. 9 The evolution of the 
availabilities for the current 
situation 
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study, the objective is the operational availability which is the most relevant, and 
therefore the MDT indicator is 60.5 hours (as being presented in [3]), respec- 
tively the repair rate is fiio — 0.01652 hours"'. For Sgnai, the failure rate will be 
^01 — 0.03846 hours~\ and the repair rate in this case is kept constant although 
certainly it will grow by introducing the redundant module. Figure 9 presents the 
comparative evolution of the availabilities, using (5). 

As being observed in Fig. 9, the consequence of implementing the redundant 
module to correct the orientation of the mobile robot through image processing is 
that the overall availability is increased by approximately 7%. 

The second experiment follows the same hypothesizes as the previous one, but 
considering the fact that the repair rate will be influenced also by the decrease 
of the MTBF indicator. The influence is obvious, but the percentage of which 
the MDT indicator is modifying is just presumed. The analysis will consider that 
the MDT will decrease to 45 hours and therefore the repair rate will became 
fj,io = 0.02222 hours"'. Figure 10 illustrates the results of this analysis: 



Fig. 10 The operational 
availability evolutions for the 
current situation 
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From Fig. 10 it can be noticed that the steady-state availability of Sgnai is bigger 
than the steady-state availability of Sinitiai by approximately 13%. 

The last experiment is focused on the inherent availability, when the MTTF in- 
dicator is 1.3 hours (value provided in [3]). The repair rate in these conditions is 
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Fig. 11 The inherent 
availabihty evolution for the 
current situation 
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fxio — 0.76923 hours"^ . The values of the failure rates are considered to be the same 
with the ones from the previous experiment. It is considered also that the repair rate 
will not be modified following the implementation of the redundant module which 
will correct the orientation of the wheeled mobile robot (this fact implies a minimal 
influence over the availability, although, certainly the repair rate will also be influ- 
enced by implementing the redundant module). Figure 1 1 illustrates the availability 
evolution for the Sinitiai system and the Sfinai system, and an inherent availability 
increase of approximately 1.5% of the Sfinai over the Sinitiai system can be noticed. 



6 Conclusions 

The paper showed two availability studies regarding the localization module of the 
wheeled mobile robots which are moving in partially known environments. The 
method illustrated and used to analyze the availability was based on Markov models. 

The first study was showing the availability evolution before and after imple- 
menting a fault tolerant controller in order to tolerate certain faults of the distance 
sensor. The second study provided the comparative availability evolution for the 
wheeled mobile robot before and after implementing a redundant method based on 
image processing designed to correct odometry errors. 

In conclusion, the necessity to analyze and increase the availability of the 
wheeled mobile robots is obvious and it is a permanent requirement in order to 
achieve a higher degree of autonomy. 
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The Use of Higher-Order Spectrum for Fault 
Quantification of Industrial Electric Motors 



Juggrapong Treetrong 



Abstract This chapter proposes a new method of electric motor fault quantifica- 
tion. Higher Order Spectrum (HOS) is a signal processing used as a fault quantifi- 
cation technique. Previous researches have shown that the faults in the stator or ro- 
tor generally show sideband frequencies around the main frequency (50 Hz) and its 
higher harmonics in the spectrum of the Motor Current Signature Analysis (MCSA). 
However in the present experimental studies such observations are not seen, but the 
faults in the stator or the rotor may distort the sinusoidal response of the motor RPM 
and the main frequency. Hence this research proposes the HOS here, namely the Bis- 
pectrum of the MCSA, because it relates to both amplitude and phase of number of 
harmonics in a signal. The Bispectrum with the unwrapped phase angle along its 
frequency is also analyzed. The tests can show that the proposed method can detect 
the faults accurately. The proposed method can also show that the severity level of 
the faults can be measured by observing the change in the heights of the Bispectrum 
amplitude. 

Keywords Induction motors ■ Higher Order Spectrum (HOS) ■ Bispectrum ■ 
Condition monitoring • Fault detection 



1 Introduction 

Induction motors are the most widely used motors among different electric motors 
because of their high level of reliability, efficiency and safety. However, these motors 
are often exposed to hostile environments during operation which leads to early 
deterioration leading to the motor failure. It has also been observed that 30^0% of 
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all recorded faults are generally related to the stator or armature faults caused due 
to the shorting of stator phase winding and 5-10% fault related to the rotor (broken 
bar and/or end ring fault) as shown by Nandi et al. [16]. Hence the condition and 
monitoring technique has generally been used to detect the fault at the early stage 
so that the remedial action can be done in much planned way to reduce the machine 
downtime and maintain the overall plant safety. 

Motor Current Signature Analysis (MCSA) is one of the most spread procedures 
for health monitoring of the motor since decades. One of the main reasons for using 
this method is that the other methods require invasive access to the motor and they 
also need extra equipment/sensors for measuring the required signals. The research 
has been progressed in mainly two directions using the stator phase current and 
voltage signals - the detection of faults and the quantification of the faults by the 
motor parameters estimation [3, 4, 12, 13, 19]. First one is important for the quick 
health assessment on routine basis, however the later one useful to know the extent 
of the faults so that remedial action can be done quickly. There are number of the 
research studies that have used the spectrum of the stator phase current signal for 
the stator motor faults [1, 5, 10, 17, 18] and the rotor fauhs [2, 6, 8, 11, 14], often 
based on the presence of the side band frequency (related to the slip frequency) and 
its harmonics around the power supply frequency or/and its harmonics. However in 
the present experiments, the side bands are not clearly seen for both rotor and stator 
faults in their spectra with frequency resolution of 1.25 Hz when using the motor 
stator phase current signals, hence the use of the Higher Order Spectrum (HOS) [7, 
9] is planned to apply for the stator phase current signal instead of the spectrum. 
Because the faults in the motor are expected to generate harmonic components of 
the motor RPM and the mains frequency in the motor current signal, hence the 
relation between different harmonic components in the signal is exploited using 
the HOS, namely the Bispectrum. The propose method is verified by testing on 
different motor fault conditions. The tests show that the proposed method can be 
useful in detection and quantification of the rotor and stator faults. This chapter is 
as a revised work from the corresponding conference paper that the reference can 
be seen from Treetrong [20]. The following sections, the concept of the Bispectrum 
is discussed and the Bispectrum results of the experimental cases of the induction 
motor with the healthy, stator winding short-circuits (stator fault) and broken rotor 
bars (rotor fault) conditions are presented. 



2 Higher Order Spectrum (HOS) 

The conventional power spectrum density (PSD) provides information on the 
second-order properties (i.e., energy) of a signal whereas the bi-spectrum can pro- 
vide information on the signal's third-order properties. In a physical sense, the bi- 
spectrum provides insight into non-linear coupling between frequencies (as it in- 
volves both amplitudes and phases) of a signal compared to the traditional PSD that 
gives only the content of different frequencies and their amplitudes in a signal. The 
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Fig. 1 Schematic of the test rig 
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Fig. 2 A typical current plot 




power spectram of a time series x(r) is computed by the discrete Fourier transform 
(DFT) of the signal as 



PSD, S,Afk)^E[X{fk)X*{fk)\, ^=l,2,3,...,Af 



(1) 



where Sxxifk) is the PSD, X(/t)and X*(fk) are the DFT and its conjugate at fre- 
quency fk for the time series x(t). N is frequency points. E{.] denotes the mean 
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Fig. 3 The spectrum plots of 
stator phase current: 

(a) healthy motor, 

(b) 15 turn's short circuit, 

(c) broken rotor bars 
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operator here. Let us assume that the time domain signal, x{t), of the time length 
equals to t. This time signal is divided into n number of segments with some over- 
lap and each segment contains 2N number of data points with sampling frequency, 
fs Hz. If Xrifk) is the FT of the rth segment, Xr{t), at the frequency, fk, then the 
averaged or mean PSD [7, 9] can be computed as 



Bispectrum, B.^Afi, fm) = E[X{fi)X{f,n)X\fi + /„)] 



B(//,/™): 



E"=i X,(/,)X,(/^)X*(/, + U) 



l + m <N 



(2) 
(3) 



The Bispectrum is complex and interpreted as measuring the amount of coupling 
between the frequencies at //, /„,, and // + /,„ and is described by 'quadratic phase 
coupling'. It is assumed that if the frequencies, // and /,„ are the pth and ^th har- 



The Use of Higher-Order Spectrum 



63 



Fig. 4 The Bispectrum plot 
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monies of the motor RPM then the component of the Bispectrum, B(fu fm) can 
represent as Bpq for better understanding. 



3 Experimental Study 

The schematic of the test rig is shown in Fig. 1. The test rig consists of an induction 
motor (4 kW, 1400 RPM) with load cell with a facility to collect the 3-phase current 
data directly to the PC at the user define sampling frequency. 

The experiments are tested on these 3 different conditions - Healthy, Stator Fault 
and Rotor Faults at different load conditions. The data are collected at the sampling 
frequency of 1280 samples/s. The motor of the stator fault type can be adjusted into 
3 differences of the stator short circuits - 5 turn shot circuit, 10 turn short circuit 
and 15 turn short circuit and the motor of the rotor fault type is broken rotor bars. 



4 Result Analysis 

A typical current plot for the healthy motor operating at 100% load is shown in 
Fig. 2. The rated current for the motor is close to 10 Amperes. 



4.1 Current Spectrum 



The amplitude spectrums are firstly applied for processing the stator current signals 
from all the experimental data of each motor condition. The frequency resolution is 
kept 1 .25 Hz with 90% overlap and number of average 82 for all the signal process- 
ing. 
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Fig. 5 The Bispectrums plots 
of the stator phase current: 

(a) 5 turns short circuit, 

(b) 10 turns short circuit, and 

(c) 15 turns short circuit 
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The computation time using the Pentium-IV PC for both the spectrum and Bis- 
pectrum is less than 30 s which it can see that the processing is definitely quick for 
the health monitoring purpose. Few typical plots for the amplitude spectrum of dif- 
ferent conditions at full load are shown in Fig. 3. The amplitude spectrum is difficult 
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Fig. 6 The Bispectrum plot 
of the stator phase current for 
the broken rotor bar motor 
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Frequency, (f„), Hz 



Table 1 Comparison of the Bispectrum component amplitudes for different fault conditions for 
the motor when operating at 100% load (unit: cubic Ampere (A^)) 





Su 


B22 


(S21) 


B24 

(B24) 


B26 

(fi62) 


Healthy 


0.025 


2.70 


0.020 


0.080 


0.018 


Faulty Rotor 


0.220 


0.28 


0.190 


0.018 


0.023 


Faulty Stator (5 turn) 


0.024 


2.60 


0.018 


0.22 


0.015 


Faulty Stator (10 turn) 


0.050 


2.30 


0.039 


0.19 


0.042 


Faulty Stator (15 turn) 


0.030 


2.00 


0.023 


0.17 


0.022 



to identify the faults with the frequency resolution of 1 .25 Hz of the current signals. 
All spectrums almost look identical. In all cases, machine RPM (Ix component) and 
its higher harmonics (2x, 3x, . . .) are present, but no side band frequency at the main 
frequency related to the slip frequency is seen to identify the fault as suggested in the 
earlier studies. Thus, it can be concluded based on the experiments that the spectrum 
is not showing any peaks other than machine RPM which indicates the faults by the 
relation between frequency and the slip frequency of the current signal. Thus, the 
identification based on side-band is not possible for the present experimental cases. 



4.2 Current Bispectrum 



The Bispectrum of stator phase currents from the 3 different motor conditions seem 
to identify the faults as seen from Figs. 4, 5, 6. 

The peaks in the Bispectrum plots are indicated by Z?i 1 , 5i2 , B21 and so on. Here 
B{\ means the relation of Ix, Ix and 2x components, B\2 {— B21) the relation of Ix, 
2x and 3x (Ix + 2x) components in a signal. In the Bispectrum, the only significant 
peak B22 can be seen for the healthy motor condition. However, for the faulty stator 
cases the amplitude of the Bispectrum component of the peak B22 decreases when 



66 



J. Treetrong 



Fig. 7 The unwrapped phase 
plots of the Bispectrum, 
(a) heakhy, (b) stator fauk - 5 
turns short circuit, (c) rotor 
fault - broken bars 
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the level of the stator faults increases and the significant increase in the amplitude of 
the Bispectrum component of the peak B24 {— B4.2), compared to the healthy condi- 
tion, nearly 2.4 times. Other peaks, Bi\, B12 {— B21), and 526 i— B^a) observed to 
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remain comparable with the amplitudes for the healthy condition (as seen in Figs. 4 
and 5). However, in the case of the faulty rotor case the peak B22 decreases signif- 
icantly compared to the healthy condition and the faulty stator cases (nearly 0.10 
times) (as seen in Fig. 6), but other peaks, B\i and B12 (— B21) increase signifi- 
cantly (nearly 8-10 times) compared to the healthy and the stator fault cases. These 
observations are also summarized in Table 1 . 

Hence based on the observation, it can be concluded that the Bispectrum of the 
phase current signal can identify and quantify the severity level of the stator and the 
rotor faults. 

The robustness of the proposed method is also confirmed by testing on differ- 
ent load levels of the motors at no load, 25% load, 50% load and 75% load for the 
healthy, the stator fault and the rotor fault cases. The tests can show that the Bispec- 
trum is consistent with the different load conditions. Additionally, the motors are 
also tested by the Bispectrum with the unwrapped phase angle. The typical phase 
plots of the method along the frequency, /„, and /; for different conditions at 100% 
load condition as shown in Fig. 7. Based on observation, the Bispectrum with the 
unwrapped phase angle can differentiate the motor faults. Different features in each 
condition case also indicate the potential use of the phase information in diagnosis. 



5 Conclusions 

The HOS, namely the Bispectrum is proposed as a new technique for motor fault 
quantification. The proposed method can show the relation between amplitude and 
phase of number of harmonics in a signal. The Bispectrum of the motor phase cur- 
rent can successfully be able to detect the stator and the rotor fault and also able to 
distinguish the stator and the rotor fault. Additionally, the method can also quan- 
tify the severity level of the faults. Thus, the proposed method can be considered 
to be a useful tool for motor fault detection and quantification. Moreover, the Bis- 
pectrum with the unwrapped phase angle along its frequency also shows a potential 
technique in condition differentiation. The future research is also planed to test the 
Trispectrum, another kind of the HOS. 
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Abstract In this paper we propose a newly multiple particle swarm optimizers with 
diversive curiosity (MPSOa/DC) for enhancing the search performance. It has three 
outstanding features: (1) Implementing plural particle swarms in parallel to explore; 
(2) Finding the most suitable solution in a small limited space by a localized random 
search for correcting the solution found by each particle swarm; (3) Introducing di- 
versive curiosity into the multi-swarm to alleviate stagnation. To demonstrate the 
proposal's effectiveness, computer experiments on a suite of benchmark problems 
are carried out. We investigate its intrinsic characteristics, and compare the search 
performance with other methods. The obtained results show that the search perfor- 
mance of the MPSOa/DC is superior to that by the PSO/DC, EPSO, OPSO, and 
RGA/E for the given benchmark problems. 

Keywords Cooperative particle swarm optimization • Hybrid computation ■ 
Localized random search ■ Exploitation and exploration ■ Diversive and specific 
curiosity • Swarm intelligence 



1 Introduction 

As a new member of genetic and evolutionary computation, particle swarm opti- 
mization (PSO) [8, 13] has been widely applied in different fields of science, tech- 
nology, and applications for solving various optimization problems [20]. This is 
because of that it has the plain advantages: intuitive understandability, ease of im- 
plementation, and distinct expressivity. Compared to genetic algorithms and evolu- 



This paper was originally presented at IMECS 2010 [26]. This is a substantially extended version. 

H. Zhang (El) 

Department of Brain Science and Engineering, Graduate School of Life Science & Systems 

Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu, Kitakyushu 808-0196, 

Japan 

e-mail: zhang@brain.kyutech.ac.jp 



S.-I. Ao et al. (eds.), Intelligent Control and Computer Engineering, 69 

Lecture Notes in Electrical Engineering 70, 

DOI 10.1007/978-94-007-0286-8_7, © Springer Science-FBusiness Media B.V. 2011 



70 H. Zhang 

tionary programming [11], information exchange, intrinsic memory, and directional 
search are the strong points of PSO. 

So far, many variants of PSO such as a particle swarm optimizer with inertia 
weight [21], a canonical particle swarm optimizer [4, 5], and fully informed particle 
swarm [14] etc. were proposed for improving the convergence and search efficiency 
of a particle swarm optimizer (the PSO). The principal objective of these optimiz- 
ers was put in methodology, i.e. enforcing the search strategy and transmission of 
information in the interior of a particle swarm. 

In recent years, many studies and investigations on cooperative PSO in relation 
to symbiosis, group behavior, and sensational synergy are in the researcher's spot- 
light. Consequently, there are different forms of cooperative PSO, for example, hy- 
brid PSO, and multi-population PSO etc. were published [1, 18] with deepening on 
group searching. In contrast to those methods that only operate a singular swarm, 
various attempts to the multi-swarm approach mainly focus on the rationality of in- 
formation propagation within plural swarms. Some reports showed that the results 
of cooperative PSO have better adaptability in optimization than uncooperative PSO 
[3, 12, 18]. 

Due to great requests to search performance and swarm intelligence, utilizing 
the techniques of group searching and parallel processing has become one of ex- 
tremely important approaches. For promoting the development of cooperative PSO 
research, we propose a newly cooperative PSO - multiple particle swarm optimizers 
with diversive curiosity (MPSOa/DC). Comparison with the convenient cooperative 
PSO, it has the following outstanding points: Decentralization in multi-swarm ex- 
ploration with hybrid search (MPSOa); Concentration in evaluation and behavior 
control with diversive curiosity (DC); And their effective combination. According 
to these, the proposal is expected to alleviate stagnation, and to enhance the search 
performance. 

The rest of the paper is organized as follows. Section 2 briefly reviews the related 
works on this study. Section 3 introduces basic algorithms of the PSO, EPSO, LRS, 
and an internal indicator. Section 4 describes the mechanism of the MPSOa/DC 
and its characteristics. Section 5 analyzes and discusses the experimental results 
to a suite of five-dimensional (5D) benchmark problems. Finally Sect. 6 gives the 
conclusions. 



2 Related Works 

Since plural particle swarm optimizers are implemented to achieve a certain goal, it 
is commonly called cooperative PSO. To all sorts of cooperative PSO, El-Abd et al. 
gave the following definition in [9] : 

Multiple swarms (or sub-swarms) searching for a solution (serially or in parallel) and 
exchanging some information during the search according to some communication strategy. 

As mentioned, it is apparent that the core technique of cooperative PSO involves 
utilizing the dispersiveness in group behaviors to function complement, and relative 
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information propagation to improve the search accuracy and efficiency. For making 
more effects, we propose the MPSOa/DC to strengthen the intelligent functions 
regarding decision making and hybrid search. 

Figure 1 illustrates the origin of the MPSOa/DC. As for uncooperative PSO, both 
of evolutionary particle swarm optimization (EPSO) [28, 29] and particle swarm 
optimization with diversive curiosity (PSO/DC) [30, 31] were published. The former 
focuses on obtaining appropriate values of parameters in the PSO by evolutionary 
computation to rationally manage the trade-off between exploitation and exploration 
in the PSO heuristic [6]. The latter pays attention to enhance swarm intelligence by 
introducing diversive curiosity into the PSO. 

As a matter of fact, the overall structure of the PSO/DC provides a good frame- 
work for methodical expansion. Needless to say, cooperative PSO also belongs 
to one of optimization methods. The proposed MPSOa/DC is an analogue of the 
PSO/DC in the curiosity-driven construction. Nevertheless, it has the following out- 
standing features: 

1. Implementing plural particle swarms in parallel to explore for enhancing the 
adaptability which effectually handles a given optimization problem. 

2. Localized random search (LRS) is separately carried out to find the most suitable 
solution in a small limited space for correcting the solution found by each particle 
swarm, respectively. 

3. Introducing diversive curiosity into the multi-swarm to check the search condi- 
tion, and to control search behavior for alleviating stagnation. 

Obviously, these strategies and the overall composition of the MPSOa/DC are 
quite different from the parallel particle swarm optimization (PPSO) [3] which only 
implements plural PSO simultaneously and the hybrid genetic algorithm and particle 
swarm optimization (HGAPSO) [12] which implements GA and PSO by the mixed 
operation. 



3 Basic Algorithms 

This section briefly describes the used PSO, LRS, and an internal indicator. 
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3.1 ThePSO 

For the sake of convenience, here, let the search space be A/^-dimensional, S2 e di'^ , 
the number of particles in a swarm be P, the position of /th particle be x' — 
{x'^,X2, . . . ,x'j^)'^ , and its velocity be v' — (v[,v'2, . . . ,v'i^)^ . In the beginning of 
the PSO search, the particle's position and velocity are generated in random, then 
they are updated by 



^k+l 



■■ CQv'f, + ciri (g) (p[ - xp + C2F2 (g) (qk - x[) 



where cq is an inertial coefficient, ci is a coefficient for individual confidence, cj is a 
coefficient for swarm confidence, ri , r2 e i){^ are two random vectors in which each 
element is uniformly distributed over [0, 1], and (g) is an element-by-element mul- 
tiplication operator. p[{— argmaxj~i kigix',)}, where g(-) is the criterion value 

of /th particle at time-step k) is the local best position of ( th particle up to now, and 
qk{— argmax,=i 2,...{g(pJ.)}) is the global best position among the whole swarm up 
to now. In the original PSO, the values of parameter, cq — 1-0 and ci — C2 = 2.0, 
are set [13]. 



3.2 TheLRS 



Random search methods are the simplest ones of stochastic optimization with undi- 
rectional search, and are effective in many problems [22]. For obtaining better search 
performance, we propose to use the LRS to find the most suitable solution from a 
limited space surrounding the solution found by the PSO. Concretely, the procedure 
of the LRS is implemented as follows. 

step 1: Let qj, be a solution found by sth particle swarm at time-step k, and set 
^now — ^k ■ Give the terminating condition, J (the total number of the LRS 
run), and set j — I- 

step 2: Generate a random data, dj e di^ ~ N(0, a^) (where Ufj is a small posi- 
tive value given by user, which determines the small limited space). Check 
whether q-^ + dj e ^ is satisfied or not. If q'^ + dj ^ ^ then adjust dj for 
moving ^| -I- dj to the nearest valid point within H. Set qnew — ql+ dj . 

step 3: Ifg(qnen') > gi^now') ^en set q^,^^, = q^ew 

step 4: Set j — j + l-lfj<J then go to step 2. 

step 5: Set q'^ — q^„g„ to correct the solution found by the sih particle swarm at 
time-step k. Stop the search. 

Due to the complementary feature of the hybrid search (i.e. memetic algorithm 
[17]), the correctional function seems to be close to the HGAPSO in search effect. 
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3.3 Internal Indicator 

Curiosity, as a general concept in psychology, is an emotion related to natural in- 
quisitive behavior for humans and animals, and its importance and effect in mo- 
tivating search cannot be ignored [7, 19]. Berlyne categorized it into two types: 
diversive curiosity and specific curiosity [2]. In the matter of the mechanism of the 
former, Loewenstein insisted that "diversive curiosity occupies a critical position at 
the crossroad of cognition and motivation" in [15]. 

Based on the assumption of "cognition" is the act of exploitation, and "motiva- 
tion" is the intention to exploration, Zhang et al. proposed the following internal 
indicator for distinguishing the above two behavior patterns [30-32]. 

yt(L,e) = max(e-2_^ — ^^,0j (2) 

where qj^i— argmax^^i ...^{^(^p}, where S is the number of plural particle 
swarms) denotes the best solution found by the whole particle swarms at time-step k. 
L is duration of judgment, and £ is the positive tolerance coefficient (sensitivity). It 
is evident that the bigger the value of the coefficient e is, the higher the sensitivity 
for exploration is, and vice verse. 



4 The IVIPSOa/DC 

Figure 2 shows a flowchart of the MPSOa/DC to illustrate the data processing and 
behavior control in the multi-swarm search. Compared to the PSO/DC [30-32], 
the most difference (yellow parts) in construction is that plural particle swarms are 
implemented in parallel, and the LRS is used to correct the solution found by each 
particle swarm, respectively. 

From the whole solutions found by the multi-swarm, the best solution, q^, is 
determined with maximum selection. Then it is put in a solution set for information 
processing. The mission of the internal indicator is to monitor whether the status 
of the best solution q^ continues to change or not. It makes up the concentration 
in evaluation and behavior control. Concretely, while the value of the output y^. is 
zero, this means that the multi-swarm is exploring the surroundings of the solution 
q^ for "cognition". If once the value of the output yj; become positive, it indicates 
that the multi-swarm has lost interest, i.e. feeling boredom, in the solution q^ for 
"motivation". 

Due to the reduction of boredom behavior of the multi-swarm search, the search 
efficiency finding an optimal solution or near-optimal solutions is drastically im- 
proved. Here, it is to be noted that the repeat of reinitialization decided by the signal 
djc in Fig. 2 is a mere expression style which concretely realizes diversive curios- 
ity to a positive search. Of course, the style is not an isolated one, it also can be 
implemented by other operation ways in practice. 
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Fig. 2 (Color online) A flowchart of the MPSOa/DC 



Table 1 Functions and criteria to the given suite of benchmark problems. The search space for 
each benchmark problem is limited to J2 e (—5.12, S.ll)*^ 



Function 



Equation 



Criterion 



Sphere 
Griewank 
Rastrigin 
Rosenbrock 



fspCx) = TL 



fCr(x)- 



4(5)0 Ell xj- nil cos(^) + l 



fRaix) =J2^^l{xj - 10cOs(237-Xd) + 10) 
Ao(5)=El"/[(100(^rf+l 



-xj)f + (xd-lf] 






In addition, the PSO estimated by EPSO [28, 29] are used in MPSOa/DC for en- 
suring higher search performance. Specifically, the genetic operations of the EPSO: 
roulette wheel selection, BLX-a crossover [10], random mutation, non-redundant 
strategy [27], rank operation, mixing operation, and elitism strategy are adopted 
[33], and a temporally cumulative fitness function of the best particle is implemented 
for evaluating the search performance of the PSO. 



5 Computer Experiments 



To facilitate comparison and analysis of the performance indexes of the proposed 
method, we use a suite of benchmark problems in Table 1 [24] . And Table 2 gives 
the major parameters employed for the next experiments. 
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Table 2 The major parameters used in the EPSO and MPSOo'/DC 

Parameters Value Parameters Value 

The number of individuals, M 10 The number of iterations, K 400 

The number of generation, G 20 The maximum velocity, Un,ax 5.12 

Probability of BLX-2.0 crossover, pc 0.5 The number of particles, P 10 

Probability of random mutation, p„, 0.5 The range of LRS, cr^ 0.05 

The number of particle swarms, 5 3 The number of LRS runs, / 10 



Table 3 


The resulting values of parameters in 


the PSO for each 5D benchmark 


problem 


Problem 


Parameters 


in the PSO 




Fitness 




CO 


Cl 


C2 





Sphere 0.677 ± 0.23 1.129 ±0.09 0.937 ±0.65 394.1 ±0.5 

Griewank 0.510 ±0.26 2.086 ±0.42 1.025 ±0.61 396.6 ±0.6 

Rastrigin 1.345 ±0.54 10.28 ±3.52 24.92 ±21.8 395.7 ±0.5 

Rosenhrock 0.902 ± 0.06 1.309±0.56 0.761 ±0.16 317.1 ±18.8 



5.1 Results of the EPSO 

Table 3 shows the resulting values of parameters in the PSO corresponding to each 
given 5D benchmark problem with 20 trials. We can observe that the average of the 
parameter values of the estimated PSO are quite different from that of the original 
PSO for the whole problems. Specially, the average of the parameter values, cq, are 
less than 1 except for the Rastrigin problem. This suggests that the active behavior 
of particles is convergence in exploring solutions. In contrast to this, the average of 
the parameter values, cq, drastically exceeds 1 indicates that the exploration needs 
to have more randomization without restriction for handling the Rastrigin problem. 
These estimated PSO in Table 3 as the PSO* are adopted in the MPSOa/DC 
for ensuring the convergence and search accuracy, and improving the search perfor- 
mance. 



5.2 Results of the MPSO*a/£>C 

For clarifying the characteristics of the MPSO*a/DC, the experiments are carried 
out by tuning the parameters, L and e. Figure 3 gives the search performance of the 
MPSO*a/DC for each problem with 20 trials. From Fig. 3, the following character- 
istics of the MPSO*a/DC are observed. 

• The average of reinitialization frequencies monotonously increases with incre- 
ment of the tolerance parameter, e, and decrement of the duration of judgment. 
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Fig. 3 Distribution of the obtained results with tuning the parameters, L and £, for each problem, 
(a) Best criterion value, (b) average of criterion values, (c) average of reinitialization frequencies 



L, for each benchmark problem. Whereas, the changes of the best criterion value 
and the average of criterion values are non-monotonous. 

• Both of the best criterion value and the average of criterion values do not change 
at all with tuning the parameters, L and e, for the Rastrigin problem. 

• For obtaining superior search performance of the MPSO*a/DC, the recom- 
mended range of the parameters, Ll e (30 ~ 90) and e^ e (10~^ ~ lO"'*) for 
the Sphere problem; L^,. e (30 ~ 50) and e^^ e (10"^ ~ lO""^) for the Griewank 
problem; L^^ e (10 ~ 90) and e^^ e (10"*^ ~ 10"^) for the Rastrigin problem; 
^*Ro ^ (40 ~ 80) and e^^ e (10~^ ~ 10~^) for the Rosenbrock problem are avail- 
able. 
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As to the Rastrigin problem, the resulting best criterion value and the average 
of criterion values are mostly unchanged with tuning the parameters, L and e. This 
result suggests that the optimized PSO* is sufficient to handle the problem. 

On the other hand, due to stochastic factor in the PSO search and complexity 
of the given problems, some irregular change of the experimental results can be 
discovered in Fig. 3 (b.Gr) and (b.Ro). Moreover, because of the better effect of the 
hybrid search, the fundamental finding, "the zone of curiosity," in psychology [7] 
is not distinguished except for the Rosenbrock problem. Hence, the surface of "the 
average of criterion values" seems to be a plane without the change of the parameters 
L and e. This result indicates that the MPSO*q!/DC has good adaptability to solve 
the given problems regardless of that the parameter L is short and e is bigger. 

We also observe that the average of reinitialization frequencies is over 300 in the 
case of the parameters, L = 10 and e — 10~^, for the Rosenbrock problem in Fig. 3 
(c.Ro). Since the average of criterion values is the lowest than that in the other 
cases in Fig. 3 (b.Ro), this result shows that the active exploring of the multi-swarm 
seems to have entered "the zone of anxiety," which leads the search performance of 
the MPSOa/DC to be lower. 



5.3 Performance Comparison 

To declare the intrinsic characteristics and the effect of the multi-swarm and hybrid 
search, the following investigation are carried out. 



5.3.1 Singular vs. Multiple Swarm Search 

For equal treatment in search, the number of particles used in a singular swarm is 
the same to the total number of particles used in the multi-swarms. Figure 4 shows 
the resulting difference, Ap5 = g* — g* (g* : the average of criterion values of the 
MPSO*a/DC, g|: the average of criterion values of the PSO*q'/DC). We can see 
that the search performance of both the MPSO*q;/DC and PSO*q;/DC seems to be 
the same for the Sphere and Rastrigin problems. This is because the simplicity of 
the Sphere problem and the effect of the EPSO to the Rastrigin problem. Based 
on the distribution of positive values of the difference, Aps, we also observe that 
the search performance of the MPSO*q!/DC is better than that by the PSO*q!/DC 
for the Griewank problem, and for the Rosenbrock problem under the condition of 
e<10-^ 

In other words, under the situation of low-sensitivity, i.e. a feeling of easiness, 
the search efficiency of plural particle swarms is superior to that by singular particle 
swarm for each benchmark problem. Accordingly, the effectiveness of implement- 
ing plural particle swarms to exploration is demonstrated well. 
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Fig. 4 The performance comparison between the MPSO*a/DC and PSO*a/DC 



5.3.2 Effect of the LRS 



For investigating the performance difference between the MPSO*q;/DC and MPSO*/ 
DC, Fig. 5 shows the obtained experimental results corresponding to same prob- 
lems. Note that the difference is defined by Ap^r = g* — g^ (g^: the average of 
criterion values of the MPSO*/DC). Similar to the preceding results, the search per- 
formance of both the MPSO*q;/DC and MPSO*/DC also seems to be the same for 
the Sphere and Rastrigin problems. 

On the one hand, the search performance of the MPSO*q;/DC is better than that 
by the MPSO*/DC under the condition of e < 10~^ for the Griewank problem. 
On the other hand, the effect of the LRS is not remarkable under the condition of 
e < 10~^ for the Rosenbrock problem. This result fits in with "no free lunch" (NFL) 
theorem [25]. 

The above results suggest that the effect of the LRS closely depends on the object 
of search, which related to how to set the parameter values for the running num- 
ber, J , and the search range, a^, and the inherent feature of the given benchmark 
problems. This is also a hot topic regarding how to rationally manage the trade-off 
between computational cost and search performance [23]. The details on discussion 
for the issue are omitted here. 
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Table 4 The mean and standard deviation of criterion values in each method for each 5D bench- 
mark problem with 20 trials. The values in bold signify the best result for each problem 



Problem 



MPSO*a!/DC PSO*/DC 



EPSO 



OPSO 



RGA/E 



Sphere 1.000 ± 0.000 

Griewank 1.000 ± 0.000 

Rastrigin 1.000 ± 0.000 

Rosenblock 0.989 ± 0.012 



1.000 ± 0.000 
1.000 ± 0.000 
1.000 ± 0.000 
0.625 ± 0.232 



1.000 ± 0.000 
0.987 ± 0.010 
1.000 ± 0.000 
0.469 ± 0.280 



1.000 ± 0.000 
0.944 ± 0.043 
0.265 ±0.118 
0.392 ±0.197 



0.998 ± 0.001 
0.796 ±0.117 
0.961 ± 0.023 
0.372 ±0.136 



5.3.3 Comparison with Other Methods 

For further illuminating the effectiveness of the proposed method, we compare the 
search performance with the other methods such as the PSO*/DC, EPSO, OPSO 
(optimized particle swarm optimization) [16], and RGA/E. 

Table 4 gives the experimental results of implementing these methods with 20 tri- 
als. It is well shown that the search performance of the MPSO*q;/DC is better than 
that by the PSO*/DC, EPSO, OPSO, and RGA/E by comparison with the average 
of criterion values. The results sufficiently reflect that the merging of both multiple 
hybrid search and the mechanism of diversive curiosity takes the active role in han- 
dling these benchmark problems. In particular, we can confirm that a big increase, 
i.e. the average of criterion values by implementing the MPSO*a/DC steeply rises 
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from 0.4694 to 0.9893, in search performance is aciiieved well for the Rosenbrock 
problem. 



6 Conclusions 

A newly cooperative PSO - multiple particle swarm optimizers with diversive cu- 
riosity, MPSOa/DC, has been proposed in this paper. Owing to the essential strate- 
gies of decentralization in search and concentration in evaluation and behavior con- 
trol, the combination of the adopted hybrid search and the execution of diversive 
curiosity, theoretically, has good capability, which greatly improves the search effi- 
ciency and effectively alleviates stagnation in optimization. 

Applications of the MPSOa/DC to a suite of 5D benchmark problems well 
demonstrated its effectiveness. The experimental results verified that unifying the 
both characteristics of multi-swarm search and the LRS is successful and effec- 
tive in convergence and adaptability. Comparison with the search performance of 
the PSO/DC, EPSO, OPSO, and RGA/E, it is confirmed that the proposed method 
has an enormous latent capability in handling different benchmark problems and 
the outstanding powers of multi-swarm search. Accordingly, the basis of the devel- 
opment study of cooperative PSO research in swarm intelligence is expanded and 
consolidated. 

It is left for further study to apply the MPSOa/DC to practical problems in the 
real- world and dynamic environments. 
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Predicting the Toxicity of Chemical Compounds 
Using GPTIPS: A Free Genetic Programming 
Toolbox for MATLAB 
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Abstract In this contribution GPTIPS, a free, open source MATLAB toolbox for 
performing symbolic regression by genetic programming (GP) is introduced. GP- 
TIPS is specifically designed to evolve mathematical models of predictor response 
data that are "multigene" in nature, i.e. linear combinations of low order non-linear 
transformations of the input variables. The functionality of GPTIPS is demonstrated 
by using it to generate an accurate, compact QSAR (quantitative structure activ- 
ity relationship) model of existing toxicity data in order to predict the toxicity of 
chemical compounds. It is shown that the low-order "multigene" GP methods im- 
plemented by GPTIPS can provide a useful alternative, as well as a complemen- 
tary approach, to currently accepted empirical modelling and data analysis tech- 
niques. GPTIPS and documentation is available for download at http://sites. google. 
com/site/gptips4matlab/. 

Keywords Genetic programming ■ Symbolic regression • QSAR ■ Toxicity 



1 Introduction 

Genetic programming [6] is a biologically inspired machine learning method that 
evolves computer programs to perform a task. It does this by randomly generating a 
population of computer programs (represented by tree structures) and then mutating 
and crossing over the best performing trees to create a new population. This process 
is iterated until the population contains programs that (hopefully) solve the task 
well. 

When the task is building an empirical mathematical model of data acquired 
from a process or system, the GP is often known as symbolic regression. Unlike 
traditional regression analysis (in which the user must specify the structure of the 
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Table 1 Selected chemical engineering applications of GP 



Authors 



Application area 



Model/application 



GreeffandAldrich[2] 
McKay et al. [9] 
Grosman and Lewin [3] 
Hinchliffe and Willis [4] 
Madar et al. [8] 
Wang and Li [17] 
Seavyetal. [14] 



Acid pressure leaching 
Distillation column 
Catalytic reaction 
Cooking extruder 
Polymerisation 
Distillation column 
Vapour liquid equilibrium 



Extent of dissolution model 
Inferential Sensor 
Reaction rate model 
Dynamic process model 
Dynamic process model 
Optimise sequence 
Hybrid model 



model), GP automatically evolves both the structure and the parameters of the math- 
ematical model. Symbolic regression has had both successful academic [1] and in- 
dustrial applications in a variety of disciplines. For instance, in the discipline of 
Chemical Engineering, an overview of selected applications of GP is given in the 
table above. 

In all the applications cited in Table 1 there is a common theme; GP is used for 
symbolic regression and, when using industrial data, the evolved models are shown 
to have better or comparable accuracy to alternative nonlinear modeling approaches 
such as neural networlcs. 

The purpose of this chapter is to introduce a free open source IVIATLAB tool- 
box called GPTIPS [12, 13] that was written for the specific purpose of performing 
symbolic regression. GPTIPS employs a unique type of symbolic regression called 
"multigene" symbolic regression [5] that evolves linear combinations of non-linear 
transformations of the input variables. When the transformations are forced to be 
low order (by restricting the GP tree depth) this, in contrast to "standard" sym- 
bolic regression, allows the evolution of accurate, relatively compact mathematical 
models of predictor - response (input - output) data sets, even when there are a 
large number of input variables. Hence, the authors believe that GPTIPS provides a 
useful, free and complementary alternative to current data analysis techniques and 
has a broad spectrum of applicability across many scientific and engineering disci- 
plines. 

This chapter is structured as follows. Section 2 provides a brief overview of GP. 
Next, Sect. 3 discusses the multigene low order GP approach that GPTIPS imple- 
ments. In Sect. 4, some of the features of GPTIPS are described. In Sects. 5-8, 
the capabilities of GPTIPS are demonstrated by using it to evolve an accurate, 
relatively compact mathematical model to predict the toxicity of chemical com- 
pounds using a data set from the literature containing over 1000 compounds along 
with measured toxicity values. Finally, in Sect. 9 we provide some concluding re- 
marks. 
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2 Genetic Programming 

The evolutionary computational (EC) method of GP evolves populations of sym- 
bolic tree expressions to perform a user specified task. A comprehensive, free to 
download introduction to GP and review of the literature is provided by Poll et al. 
[10] but a brief description of GP is provided here. 

In GP, each tree expression can be thought of as being analogous to the DNA of 
an individual in natural evolution. The evolution of the expressions occurs over a 
number of generations (iterations) and each new generation of individuals is created 
from the existing population by direct copying as well as performing operations on 
the individuals analogous to the alterations to DNA sequences that naturally occur 
during sexual reproduction and mutation. This is accomplished by evaluating each 
individual in the current population to determine its 'fitness' (i.e. its performance 
on the user specified objective function or functions) and performing probabilistic 
selection and recombination of individuals biased towards those that are relatively 
fit compared to the other individuals in the population. 

At the beginning of each run, a population of symbolic expressions is randomly 
generated. This is accomplished using a simple tree building algorithm that ran- 
domly selects nodes, with replacement, from a pool comprising primitive func- 
tions (e.g. addition, subtraction, the hyperbolic tangent, natural logarithm, expo- 
nential, etc.), the input variables as well as randomly generated constants. These 
nodes are randomly assembled into tree structured symbolic expressions, subject 
to user-defined tree size and/or depth constraints. After evolving the population for 
a number of generations by copying, mutation and recombination operations, the 
tree expression with the best fitness is usually selected as the best solution to the 
problem. 

Two principal genetic recombination operators are used in GP: sub-tree crossover 
and sub-tree mutation. Sub-tree crossover is an operation performed on two parent 
trees that generates two offspring. For each expression, a sub-tree is randomly se- 
lected. These sub-trees are then exchanged to create two new expressions to go into 
the next generation. Sub-tree mutation operates on a single parent expression and 
generates a single offspring expression. First, a randomly selected sub-tree of the 
parent is deleted. Then, a new sub-tree is randomly generated using the same tree 
building algorithm that was used to build the initial population of expressions. The 
resulting offspring expression is then inserted into the new population. The mutation 
operation is used relatively infrequently compared to the crossover operation and its 
purpose is to maintain genetic diversity over the course of the run and to prevent 
premature convergence to unsatisfactory solutions. 

In GP, the choice of the primitive functions is domain dependent and in symbolic 
regression, where there is little or no prior knowledge of the underlying relation- 
ships, mathematical operators such as those described above are typically employed 
with a high degree of success [9, 13]. In practice, it is often best to perform some 
initial runs with a few simple primitives (e.g. addition, multiplication and subtrac- 
tion) and then incrementally add other non-linear primitives - such as the hyperbolic 
tangent function - to evaluate whether more accurate and compact symbolic expres- 
sions may be evolved. 
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y = do + cy, (0.4 1X1 + tanh(X2X3) + 0/2(0.45X3 + sqrt(X2)) 
Fig. 1 Example of a multigene symbolic model 



3 Multigene Symbolic Regression 

Typically, symbolic regression is performed by using GP to evolve a population 
of trees, each of which encodes a mathematical equation that predicts st (N x 1) 
vector of outputs y using a corresponding (A^ x M) matrix of inputs X where A^ is 
the number of observations of the response variable and M is the number of input 
(predictor) variables. I.e. the /th column of X comprises the N input values for the 
/ th input variable and may be designated as the input variable x, . 

In contrast, in multigene symbolic regression each symbolic model (and each 
member of the GP population) is a weighted linear combination of the outputs from 
a number of GP trees, where each tree may be considered to be a "gene". For exam- 
ple, the multigene model shown in Fig. 1 predicts an output variable y using input 
variables X\,X2 and X3. 

This model structure contains non-linear terms (e.g. the hyperbolic tangent) but is 
linear in the parameters with respect to the coefficients dQ, d\ and d2- In practice, the 
user specifies the maximum number of genes Gmax a model is allowed to have and 
the maximum tree depth Dmax any gene may have and therefore can exert control 
over the maximum complexity of the evolved models. In particular, we have found 
that enforcing stringent tree depth restrictions (i.e. maximum depths of 4 or 5 nodes) 
often allows the evolution of relatively compact models that are linear combinations 
of low order non-linear transformations of the input variables. 

For each model, the linear coefficients are estimated from the training data us- 
ing ordinary least squares techniques. Hence, multigene GP combines the power of 
classical linear regression with the ability to capture non-linear behaviour without 
needing to pre-specify the structure of the non-linear model. In Hinchliffe et al. [5] 
it was shown that multigene symbolic regression can be more accurate and com- 
putationally efficient than the standard GP approach for symbolic regression and 
Searson et al. [13] demonstrated that the multigene approach could be successfully 
embedded within a non-linear partial least squares algorithm. 

In GPTIPS, the initial population is constructed by creating individuals that con- 
tain randomly generated GP trees with between 1 and Gmax genes. During a GPTIPS 
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run, genes are acquired and deleted using a tree crossover operator called two point 
high level crossover. This allows the exchange of genes between individuals and it is 
used in addition to the "standard" GP recombination operators. If the ; th gene in an 
individual is labelled G, then a two point high level crossover is performed as in the 
following example. Here, the first parent individual contains the genes (G1G2G3) 
and the second contains the genes (G4G5G6G7) where G^ax = 5. Two randomly 
selected crossover points are created for each individual. The genes enclosed by the 
crossover points are denoted by < . . . >. 

(Gi < G2 > G3) (G4 < G5G6G7 >) 

The genes enclosed by the crossover points are then exchanged resulting in the two 
new individuals below. 

(G1G5G6G7G3) (G4G2) 

Two point high level crossover allows the acquisition of new genes for both indi- 
viduals but also allows genes to be removed. If an exchange of genes results in an 
individual containing more genes than Gmax then genes are randomly selected and 
deleted until the individual contains Gmax genes. 

In GPTIPS, standard GP subtree crossover is referred to as low level crossover. 
In this case, a gene is selected randomly from each parent individual, standard sub- 
tree crossover is performed and the resulting trees replace the parent trees in the 
otherwise unaltered individual in the next generation. GPTIPS also provides several 
methods of mutating trees. 

The user can set the relative probabilities of each of these recombinative pro- 
cesses. These processes are grouped into categories called events. The user can then 
specify the probability of crossover events, direct reproduction events and mutation 
events. These must sum to one. The user can also specify the probabilities of event 
subtypes, e.g. the probability of a two point high level crossover taking place once 
a crossover event has been selected, or the probability of a subtree mutation once 
a mutation event has been selected. However, GPTIPS provides default values for 
each of these probabilities so the user does not need to explicitly set them. 



4 GPTIPS Features 

GPTIPS is a predominantly command line driven open source toolbox that requires 
only a basic working knowledge of MATLAB. A run is configured by a simple con- 
figuration M file and there are a number of command line functions to facilitate post- 
run analyses of the results. Whilst not an exhaustive list, GPTIPS currently contains 
the following configurable GP features: tournament selection & plain lexicographic 
tournament selection [7], elitism, three different tree building methods (full, grow 
and ramped half and half) and six different mutation operators: (1) subtree mutation 
(2) mutation of constants using an additive Gaussian perturbation (3) substitution of 
a randomly selected input node with another randomly selected input node (4) set a 
randomly selected constant to zero (5) substitute a randomly selected constant with 
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another randomly generated constant (6) set a randomly selected constant to one. 
In addition, GPTIPS can, without modification in the majority of cases, use nearly 
any built in MATLAB function as part of the function set for a run. The user can 
also write bespoke function node M files and fitness functions; hence GPTIPS can 
be used to solve problems other than non-linear modeling/symbolic regression. 

In addition, GPTIPS has a number of features that are specifically aimed at 
the creation, analysis and simplification of multigene symbolic regression mod- 
els. These include: (1) use of a 'holdout' validation set during training to mitigate 
the effects of overfitting (2) graphical display of the results of symbolic regression 
for any multigene model in the final population (3) mathematical simplification of 
any model (4) conversion to LaTeX format of any model (5) conversion to PNG 
(portable network graphics) file of the simplified equation of any model (6) con- 
version of any model to standalone M file for use outside GPTIPS (7) graphical 
display of the statistical significance of each gene in a model (8) functions to re- 
duce the complexity of any model using "gene knockouts" to explore the trade off 
of model accuracy against complexity (9) graphical population browser to explore 
the trade off surface of complexity/accuracy (10) graphical input frequency analysis 
of individual models or of a user specified fraction of the population to facilitate the 
identification of input variables that are relevant to the output. 

The Symbolic Math toolbox (a commercial toolbox available from the vendors 
of MATLAB) is required for the majority of the post run simplification and model 
conversion features and the Statistics Toolbox is required for the display of gene 
statistical significance. The core functionality of GPTIPS and the ability to evolve 
multigene models does not, however, require any specific toolboxes. 



4.1 Using GPTIPS for Symbolic Regression 

An example of a simple configuration file for multiple gene symbolic regression is 
shown in Fig. 2. It is assumed that data is located in the current directory in the file 
mydata.mat and comprises the training input data variable (xtrain), the training out- 
put variable (ytrain) as well as a testing data set (xtest and ytest). The data should be 
arranged by columns, e.g. the nth column of xtrain should contain the observations 
of the Mth input variable. 

In this example, only a few GPTIPS settings are specified. Any user parameters 
not explicitly set automatically use the default values. However, the user must at 
least specify the fitness function, the input and output data and the function nodes 
to be used in their configuration file. This configuration file first sets population size 
= 100 and number of generations = 100. The fitness function is then specified (i.e. 
the name of the M file) as is the fact that this is an error minimisation problem. 

Next, the user data file mydata.mat is loaded and the variables within this file 
(here called xtrain, ytrain, xtest and ytest but they could be called anything) are as- 
signed. After this is the number of input variables is set as the number of columns in 
the inputs training data matrix. Next, multigene mode is enabled and the maximum 
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function gp = mY_conf ig (gp) ; 

gp.runcontrol .pop_size = 100; 

gp. runcontrol .num_gen = 100; 

gp. fitness . fitfun = @regressinulti_f itfun; 

gp. fitness .minimisation = true; 

load mydata 

gp.userdata .xtrain = xtrain; 
gp.userdata. ytrain = ytrain; 
gp.userdata. xtest = xtest; 
gp.userdata. ytest = ytest; 

gp. nodes . inputs .num_inp = size (gp .userdata .xtrain, 2 ) 

gp. genes .multigene = true; 

gp. genes .max_genes = 4; 

gp. treedef .max_depth = 5; 

gp. nodes . functions .name = {' times ', 'minus ', 'plus '} ; 



Fig. 2 Example GPTIPS configuration file for multigene symbolic regression 

number of genes per individual is set to 4. The maximum tree depth is then set to 5. 
Finally, the function nodes times, minus and plus are specified. 



5 Evolution of a Predictive Model of Aqueous Chemical Toxicity 
Using GPTIPS 

In the remainder of this article we will demonstrate how we have used GPTIPS to 
evolve a predictive QSAR model of aquatic toxicity for chemical compounds, based 
on their molecular structure. 

QSAR (Quantitative Structure Activity Relationships) is a well established tech- 
nique for deriving structure property relationships for chemical compounds that can 
be used to predict the properties of novel chemical structures. Chemical compounds 
can be represented by a large number of computed numerical values, called "de- 
scriptors", each of which in some way characterises the structure or behaviour of 
the compound. The idea of QSAR is to build empirical or semi-empirical models 
that relate the descriptors of a compound to some physical, chemical or biologi- 
cal property. A number of software packages are available to compute descriptor 
values for compounds with a known structure. Many of these are commercial prod- 
ucts (e.g. DRAGON) but there are also free/open source packages e.g. the Chemical 
Development Kit [15]. 

A QSAR modelling scenario involves a data set of known chemical compounds 
and a measured endpoint for each compound. The measured endpoint is the prop- 
erty of interest. Typical properties of interest are those related to pharmaceutical 
drug development. These include biological activities representing the ability of 
a drug candidate to perform its desired function (e.g. IC50, the concentration of 
a compound required to inhibit a particular biological or biochemical function by 
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half) and the ADME properties (adsorption, distribution, metabolism and excretion) 
which characterise the behaviour of a of a pharmaceutical drug compound within 
the organism. 

The prediction of chemical toxicity is another chemical property that is of vital 
importance in both pharmaceutical drug development and managing the environ- 
mental risk of chemical compounds. In the latter case there are legal regulatory 
structures (e.g. the REACH regulations in the European Union - EC 1907/2006) 
that specify that QSAR models should play a part in managing this risk in order 
to reduce the costs of experimental toxicity measurement. Hence, the development 
of effective QSAR modelling methods continues to present a very real and relevant 
challenge. 

There are a number of strategies & protocols for experimentally evaluating chem- 
ical toxicity. One commonly accepted method is the measurement of the growth in- 
hibition of ciliated protozoan T. pyriformis [18]. There are freely available aquatic 
toxicity data for more than 1000 compounds, due to the efforts of Schultz and col- 
leagues [11]. Zhu et al. [18] have used this to compile a data set of 1093 unique 
compounds and have developed a number of predictive QSAR models using vari- 
ous descriptor packages and modelling methodologies. Here, the use of GPTIPS to 
evolve a predictive model of chemical toxicity using this data set is demonstrated 
(using the descriptors from the commercial DRAGON package) and the results com- 
pared with those published in Zhu et al. [18]. 



6 Data 

The T. pyriformis toxicity values (i.e. the response y data) are measured as the loga- 
rithm of the 50% growth inhibition concentration log(IGC50~^). The data available 
for training QSAR models contains 644 compounds and another 449 compounds 
are used an external test/validation data set to verify the predictive ability of the 
models. For each compound 1664 DRAGON descriptor values are used as the pre- 
dictor data (i.e. the input X data contains 1664 input variables) - compound struc- 
tures, toxicity and descriptor values are available from the EU CADASTER website 
at http://www.cadaster.eu/node/65. To mitigate against the effects of overfitting, 128 
compounds (approximately 20%) in the training data set were randomly selected for 
use as a holdout validation data set leaving the training data containing 516 com- 
pounds. In GPTIPS, holdout validation is performed as follows: at the end of each 
generation, the "best" individual (as evaluated on the training data) is then evaluated 
on the holdout validation set. The individual that performs best on the holdout set 
(over the course of the run) is stored and may be accessed after the run. 



7 GPTIPS Run Settings 

A GPTIPS run with the following settings was performed: Population size — 500, 
Number of generations = 500, Tournament size =12 (with lexicographic selection 
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pressure), Z)max = 4, Gmax — 8, Elitism = 0.01% of population, function node set 
= {plus, minus, times, tanh, sin}, terminal node set — {1664 DRAGON descriptors 
xi — JC1664, ephemeral random constants in the range [—1010]}. The default GPTIPS 
multigene symbolic regression function was used in order to minimise the root mean 
squared prediction error on the training data. 

The following (default) recombination operator event probabilities were used: 
crossover events = 0.85, mutation events = 0.1, direct reproduction — 0.05. The 
following sub-event probabilities were used: high level crossover — 0.2, low level 
crossover — 0.8, subtree mutation = 0.9, replace input terminal with another random 
terminal — 0.05, Gaussian perturbation of randomly selected constant = 0.05 (with 
standard deviation of Gaussian = 0.1). These settings are not considered 'optimal' 
in any sense but were based on experience with modelling other data sets of similar 
size. The run took approximately 15 minutes on a PC with a dual core processor 
running at 2.2 GHz with 3.5 GB of RAM. 



8 Results 

The model that performed best on the holdout validation data was chosen. This 
model has coefficients of determination (i.e. proportion of the variation in the re- 
sponse explained by the model) of R- (training) = 0.83, R^ (holdout) — 0.78 and 
R^ (test) = 0.78. In Zhu et al. [18] the results are reported in terms of MAE (mean 
absolute error) for two test tests referred to in the paper as Validation set 1 (339 com- 
pounds) and Validation set 2 (1 10 compounds) that comprise the whole test set used 
here. In terms of MAE, the evolved GPTIPS model has MAE(training) = 0.3292, 
MAE(holdout) = 0.3573 and MAE(test) = 0.3518. 

Zhu et al. [ 1 8] report the results of a number of individual models, built using var- 
ious descriptor packages and modelling techniques. Some of these models consider 
the "applicability domain" (AD) of the compounds (i.e. whether the compounds lie 
in the region of descriptor space deemed to be suitable for generating a prediction) 
whereas others do not employ AD considerations. In general, models that consider 
AD give more accurate predictions but only the results of the non AD models using 
the DRAGON descriptors are repeated here. The first DRAGON descriptor based 
model is a support vector machine [16] regression that yields MAE( Validation set 
1) = 0.37 and MAE(Validation set 2) = 0.42. This corresponds to an MAE(test) 
— 0.38. The second DRAGON based model is a /c-nearest neighbour (A;-NN) ap- 
proach that achieves MAE(Validation set 1) = 0.29, MAE(Validation set 2) = 0.43 
corresponding to MAE(test) = 0.32. Hence it can be seen that the evolved GPTIPS 
model lies between the SVM and the A:-NN approaches, i.e. GPTIPS can achieve 
predictive performance of the order of the current state of the art empirical mod- 
elling methodologies. 

GPTIPS was used to mathematically simplify and export the evolved model as a 
PNG graphics file. This is shown in Fig. 3. 

It can be seen that the evolved model is reasonably compact, consists of both 
linear terms and low order non- linear transformations of the inputs and has selected 
a small number of descriptors from the 1664 available. 
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y = -2.092 - 0.7548x911 +0.7548xi558 - 0.8997 tanh(tanh(xi426)) 

+ 0.09443(X9ii -Xi558)X654- 0.1481X1552 

+ 0.1481x391 -0.2489x1429- 0.2489 sin(x967 -X709) 
+ 0.7143x1245 -0.5978x1662- 0.5978 tanh(xi429+X525) 
+ 0.7802x1426 + 0.7802x1563 

Fig. 3 Graphical rendering of evolved symbolic T. pyriformis toxicity model 

9 Conclusions 

In this article we iiave introduced the multigene symbolic regression capabilities 
of GPTIPS and demonstrated it with an application in which a predictive symbolic 
QSAR model of T. pyriformis aqueous toxicity was evolved. It was demonstrated 
that the evolved model is compact and offers similar high performance to recently 
published QSAR models of the same data. The point of this article is not to assert 
that multigene symbolic regression (using low order non-linear transforms of the 
inputs) is better or worse than other methods, but that it is an alternative and com- 
plementary approach to existing empirical modelling and data analysis techniques. 
It is also an approach that is facilitated by the free GPTIPS toolbox for MATLAB, 
a program that is used widely in academia and industry. 
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Diversity-Driven Self-adaptation 
in Evolutionary Algorithms 
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Abstract Pareto-based multi-objective optimization problems (MOPs) are cur- 
rently best solved using evolutionary algorithms. Nevertheless, the performance 
of these nature-inspired stochastic search algorithms still depends on the suitabil- 
ity of their parameter settings with respect to specific optimization problems. The 
tuning of the parameters is a crucial task which concerns resolving the contrary 
goals of convergence and diversity. To address this issue, we propose a diversity- 
driven self-adaptive mechanism (SAM) for the simulated binary crossover. This 
novel technique exploits and optimizes the balance between exploration and ex- 
ploitation during the evolutionary process. This "explore first and exploit later" 
approach is addressed through the automated and dynamic adjustment of the dis- 
tribution index of the simulated binary crossover (SBX) operator. We conducted 
a series of experiments where SAM is applied to the Non-dominated Sorting Ge- 
netic Algorithm to solve the Sphere, Rastrigin, and ZDT optimization problems. 
Our experimental results have shown that our proposed self adaptation mecha- 
nism can produce promising results for both single and multi-objective problem 
sets. 

Keywords Self-adaptive ■ Parameter tuning ■ Simulated binary crossover ■ 
Evolutionary algorithm 



1 Introduction 

Evolutionary algorithms can efficiently solve multi-objective optimization problems 
(MOPs) by obtaining diverse and near-optimal solution sets. Multiple evolutionary 
techniques have been proposed for MOPs. Among them. Non-dominated Sorting 
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Genetic Algorithms II (NSGA-II) [6] and Strength Pareto Evolutionary Algorithm 
II (SPEAII) [13] are commonly regarded as the state-of-the-art multi-objective evo- 
lutionary algorithms (MOEAs). 

In MOEAs, crossover and mutation operators are typically utilized to produce 
offspring solutions from selected parent individuals. Both operators involve param- 
eters which dictate: 

1. The frequency (crossover and mutation rate) of the evolutionary operations. 

2. The spread (crossover and mutation distribution index) of offspring solutions. 

Both the frequency and spread properties govern the conflicting convergence and 
diversity dynamics of the evolutionary process. Consequently, the performance of 
MOEAs depends on the suitability of the above parameters setting with respect to 
specific optimization problems. The tuning of these parameters is thus a critical 
time-consuming optimization process. As a result, this limits the applicability of 
MOEAs to provide decision support for real life problems. To address this issue, 
we propose a novel self-adaptive mechanism (SAM) which aims at improving the 
MOEA's performance (when applied to different optimization problems) through 
automatically adjusting/balancing the exploration and exploitation of candidate so- 
lutions during the evolutionary search. SAM can dynamically adjust the distribution 
index of SBX operator in NSGA-II. Identifying a suitable distribution index (rjc) en- 
ables NSGA-II to optimize the balance between exploration and exploitation during 
the different stages of the evolutionary search. 

The essential idea of SAM is that if the diversity running performance is poor, 
strong evolutionary operation should be applied to break the clusters of candidate 
solutions and vice versa. Also, the crowding distance is an estimate of the surround- 
ing density of a given solution point and it could be regarded as a criterion to deter- 
mine the value of this solution. Hence, if the crowding distance is relatively high, 
soft evolutionary operation is required to preserve the solution points. 

The remainder of the paper is structured as follows: A description of related 
work is first presented. This is followed with an introduction to the SBX operator 
and diversity running performance metric. Then, a detailed description of the self- 
adaptive mechanism is provided. A series of experiments involving single and multi- 
objective optimization problems are conducted and discussed. Our conclusion and 
future work are then finally outlined. 



2 Related Work 

Past studies [7, 1 1] have proven the efficiency of the "explore first and exploit later" 
concept which relies on the intensive exploration of candidate solutions during the 
early stage and local fine-tuning during the later/final stage of the search. 

To exploit this concept in MOEAs, several self-adaptation approaches have 
been proposed [1, 10, 11]. Utilizing the feedback from the search, several adap- 
tive parameter control mechanisms were used to obtain a smooth navigation over 
the search space. For instance, in [1] a self-adaptive Pareto Differential Evolution 
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(PDE) algorithm was proposed which self-adapts the crossover and mutation rate. 
A deterministic-scheduled decreasing mutation rate was defined in [11] which also 
implemented an adaptive variation operator that facilitated the exchange of search 
information in MOPs. These self-adaptation approaches demonstrated significant 
improvements over static counterparts; note that these methods focused on the ef- 
fects of changing the crossover/mutation rates (i.e., frequency) instead of the distri- 
bution index parameter (i.e., spread). The detailed taxonomy of self-adaptation of 
the crossover/mutation rates can be found in [9]. Here we propose a complementary 
investigation examining the effects of the spread property. 

To our knowledge, the only significant reported study addressing spread was car- 
ried out in [5], in which a self-adaptive SBX (SA-SBX) was introduced to dynami- 
cally adjust (at each generation) the distribution index of SBX in NSGA-II. SA-SBX 
was found to produce better results on both single and multiple objective optimiza- 
tion problems compared to the SBX with fixed value of the distribution index. Nev- 
ertheless, a drawback of SA-SBX is that it requires another critical user-predefined 
parameter a. According to the experiments reported in [5], SA-SBX would out- 
perform the traditional non self-adaptive counterpart only when a is "well tuned" 
manually. Although the Deb et al.'s approach demonstrated better performances, 
their method introduced an additional difficulty in the already complex parameter 
tuning process. Consequently such approaches do not resolve the robustness and 
applicability issues of MOEAs. 

In contrast with Deb et al.'s approach, we propose a self-adaptive method which 
does not introduce another critical parameter to be predefined by the user. This self- 
adaptive mechanism is presented in the next section. 



3 Self-adaptive Mechanism 

The working principles of SBX are described to emphasize the importance of dis- 
tribution index r\c in generating the offspring solutions. Then, the implementation 
details of the diversity running performance metric are presented and the concept 
of crowding distance is introduced. Finally, we present the self-adaptive mecha- 
nism (SAM) which can dynamically adjust the distribution index in SBX using the 
feedback information from both the diversity running performance metric and the 
crowding distance. 



3.1 Simulated Binary Crossover (SBX) 

The SBX crossover operator [2] creates two offspring solutions (represented as real 
values) from two selected parent solutions. The procedure of deriving offspring so- 
lutions Xj- ' and Xf ' from the parent solutions x^ ' and x,- is as follow. 



98 



F. Zeng et al. 



Fig. 1 The probability 
density function for creating 
offspring solutions with the 
SBX operator (adapted from 
[2]) 
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A random number m e [0, 1] is generated. Given a pre-specified probability dis- 
tribution function (Eq. 1), the value of /?, (mathematical definition of j6, , see Eq. 8) 
is determined so that the area under the probability curve from zero to /i, is equal 
to M. The distribution index rjc is a non-negative real number. Figure 1 illustrates 
the probability density function for creating offspring solutions using the SBX op- 
erator from two example parents x^ —3 and x^ —6 with distribution index of 
r]c — 2.0 and rjc — 5.0. Larger values of r]c are more likely to produce "near parent" 
solutions whereas smaller values of rjc lead to a more diverse search. After obtaining 
Pi from Eq. 2, the offspring solutions are calculated using Eqs. 3 and 4. 
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3.2 Diversity Running Performance Metric 



A modified diversity running performance metric is implemented to dynamically 
assess the diversity performance of the generated solution sets. This diversity run- 
ning performance metric is based on the running performance metrics proposed in 
[3]. Two principal modifications are introduced: 
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Table 1 Mapping table to 
assign a value to m ( ) 
(adapted from [3]) 
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1. The number of grids (approximating the diversity of the population, see Fig. 2) 
is derived by dividing the population size by the number of objectives (instead of 
requiring the user to manually define it). 

2. Deb et al.'s approach is limited by the requirement of a priori knowledge of the 
target solutions distribution. Using this information, the number of grids can be 
determined/fitted. Nevertheless in real life optimization problems, this informa- 
tion is usually unavailable. Here the running metric does not refer to any pre- 
specified target set of solution points. Instead the running metric is employed to 
converge towards an ideal target set of solutions where each grid would possess 
a representative solution point. 

Given the minimal and maximal boundary values, the hyperplane is thus divided 
into a number of grids (population size divided by the number of objectives). The 
diversity performance metric is based on whether each grid contains a solution point 
or not. The best diversity performance is achieved if all grids contain at least a 
solution point. The steps to calculate the diversity are as follows. 



• 



Step 1 : Calculate diversity array. The number of integer variables in the diversity 
array is equal to the number of grids in the hyperplane. Each variable in the diver- 
sity array corresponds to one particular grid /. The value h{i) of the /th elements 
is derived using Eq. 5. 



Hi) 



1 , if grid / contains a representative point 
0, otherwise 



• Step 2: Assign a value, m ( ) to each grid i depending on its neighboring grids h() 
values in the diversity array. The value of the /th grid is calculated as shown in 
Table 1. 

For example let us consider the grid patterns p\ — 010 (i.e., h{i — 1) = 0, 
h{i) — 1 and h{i -|- 1) = and p2 — 101. According to Table 1, we obtain 
m{pi) — m{p2) — 0.75 which represent a good periodic spread pattern. Whereas 
if we consider pi, — 110, we obtain m{pT,) — 0.67 meaning that p^ covers a 
smaller spread. 
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Fig. 2 Example of 
computing the diversity 
metric 
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Step 3: For each objective, calculate the diversity measure d„ 
m() values. 

^number of grids _ 
d,r 



by averaging the 



Er 



'm(h(i -l),hii),h(i + l)) 



(6) 
Number of Grids 

To illustrate the procedure to calculate the diversity measure, an example is pre- 
sented in Fig. 2. 

In this example, a two-objective (/i and /2) minimization problem is exam- 
ined. The solution points are marked as points. The f2 — plane is used as the 
reference plane. Suppose the population size is 10, we divide the range of fi val- 
ues into 10/2 — 5 grids. Then, for each grid, the value of h() is calculated based 
on whether the grid contains a representative solution point or not. Then, the value 
of m() and the diversity measure are calculated based on a sliding window con- 
taining three consecutive grids. The h{) values of the imaginary boundary grids 
are always 1 as shown in the shaded grids. 

Step 4: Calculate overall diversity performance metric by averaging the diversity 
measures of all objective spaces. 

E Number of Objectives , ,.^ 
dm (l ) 

uiversity metric — -, -, (7) 



Number of Objectives 

Figure 3 illustrates the running diversity metric obtained from two example ex- 
perimental runs. For ZDTl, after the 100th generation, the diversity metric oscil- 
lates around a value of 0.85. In ZDT6 case, this diversity metric reaches steady 
state after 160 generations. Similar observations have been reported in [8]. In our 
implementation, this diversity running performance metric is used to return feed- 
back about the search space. Once the diversity metric stabilizes (i.e., when the 
exploration phase terminates) the exploitation phase may initiate. 
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Fig. 3 Diversity metric 
dynamics for ZDTl and 
ZDT6 using NSGA-II from 
two example experimental 
runs using the following 
parameters: population size = 
100, crossover distribution 
index rjc = 20.0, mutation 
distribution index )?,„ = 50.0, 
crossover probability 
Pc = 1.0, mutation 
probability p„, = 1/30 and 
1 / 10 for the benchmark 
problems ZDTl and ZDT6 
respectively, and maximum 
number of generations 
g = 500 
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3.3 The Crowding Distance 

The crowding distance indicator was proposed in [6]. It serves as an estimation of 
the size of the largest cuboid enclosing the solution point. 

It could be regarded as a criterion to determine the value of the solution point. 
In this scheme, "boundary solutions" or highest and lowest objectives are given the 
maximum value in order to retain them. The crowding distance can be calculated by 
measuring the distance between the two immediate neighbors of a given point along 
each of the objective dimensions. Lastly, the "final crowding distance" is computed 
by adding the crowding distances obtained for each objective. 



3.4 Diversity-Driven Self-adaptive SBX 



In most applications of NSGA-II, the crossover and mutation distribution index r]c 
and ri,„ are fixed. Specifically, a fixed value of rjc — 2.0 is typically chosen for 
single-objective optimization problems [4] whereas rjc — 20.0 is commonly used 
for ZDT benchmark problem sets. Although using a fixed value of rjc can also lead 
to the implementation of self-adaptive techniques, past studies using the SBX oper- 
ator with fixed distribution index could not solve multi-modal problems such as the 
Rastrigin's function [2]. 

We suggest a self-adaptive mechanism to dynamically update rjc. Here we as- 
sume that for MOPs, the optimal diversity performance could only be achieved 
when the solution set is close to the optimal solution set. Hence, if optimal di- 
versity performance is achieved, the distribution index rj/. should be large enough 
to make the offspring solutions very similar to their parents. On the other hand, if 
the diversity performance is poor, strong crossover operation should be applied to 
break the clusters of solution points. In the beginning stage of the search process. 
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Fig. 4 Mapping between /i; 
and u value in SAM 
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relatively low diversity metric results in strong crossover operation to explore the 
search space and in the later stage, soft crossover operation is applied to exploit 
local near-optimal solutions. Thus, this diversity-driven SAM can effectively ex- 
ploit the concept of "explore first and exploit later". Also, a large crowding distance 
means that the surrounding density of the solution point is low, consequently soft 
crossover operation should be applied to preserve it. 
The above SAM algorithm is now detailed: 

• Step 1 : Calculate the diversity running performance metric. 

• Step 2: Derive the reference crossover distribution index ric based on the diversity 
performance. 

The spread /J,- of the offspring solution points with respect to the parent points 
is obtained in Eq. 8. Based on /i,, crossover can be classified into three classes, 
namely contracting crossover (/i, < 1), stationary crossover (/S,- — 1), and expand- 
ing crossover (/S, > 1). The expanding crossover can "expand" the parent points 
to form more diverse offspring points. Contracting crossover has the opposite ef- 
fect of contracting the parent points. We define the value range of /J, from 0.9 to 
1.1 as the close value range (CVR) where the generated offspring solutions are 
considered very similar to parent solutions. This range was determined based on 
parametric studies carried out in [12]. 



2,r+l 



l,r+l 



2,t 



A.t 



(8) 



Here we determine the reference distribution index rjc such that the probability of 
Pi falling into the CVR (i.e., /J, e [0.9, 1.1]) equals to the diversity performance 
metric as illustrated in Fig. 4. For example, if the diversity running performance 
metric is 0.70, then we should make sure that 70% of the time Pi e [0.9, 1.1]. 
By mapping the random number u to Pi (using Eq. 2), we have u e [0.15, 0.85]. 
Then rjc can be calculated using Eq. 9: 



'7c = 



log2» 
log A 

-(H 



- 1, 

log 2(1- 



■«) 



log ft 



if < 0.5 
otherwise 



(9) 



The distribution indexes rjc — 10.42 and rji- — 11.63 are averaged and we obtain 
a reference crossover distribution index ?7c = 1 1 -0 to produce offspring solutions. 
Randomly initialized population causes poor diversity performance at the be- 
ginning and consequently lowers the probability of /J, falling into CVR. In the 
later stage, the diversity performance stabilizes at a relatively higher value and 
the exploitation phase starts as the probability of Pi in-between CVR is higher. 
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Fig. 5 Flowchart of the 
self-adaptive mechanism for 
SBX 
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Step Two: 
Determine Reference Crossover Distribution Index 
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Step Three: 

Derive Individual Crossover Distribution Index 



• Step 3: According to the crowding distances cd of the selected parents, individual 
crossover distribution indexes are assigned to improve the efficiency and accuracy 
of the crossover operator. 

For each generated offspring solution, individual crossover indexes are com- 
puted using the expression below. 

cdi+cd2 
r]c = r]cX — — =- (10) 

2 X cd 

where cdi and cdj are the crowding distances of the two selected parents and 
cd is the average crowding distance of the entire population. As devised in the 
crowding distance scheme, the boundary solutions have maximum values. Con- 
sequently these values are not included in the calculation of the average crowding 
distance. Instead, offspring solutions having boundary solutions as parent points 
are assigned with the highest distribution index to retain them. Following the pre- 
vious example, rjc — 11.0 and the crowding distances of the two parents of off- 
spring solution are 0.65 and 0.95 respectively with an average crowding distance 
of 0.50. Given Eq. 10, we have: r]^ = n.Ox\^ = 17.6. 

The above step are summarized in Fig. 5. 



4 Empirical Results 

We now apply the self-adaptive mechanism in NSGAII to two single-objective 
functions (Sphere and Rastrigin's Function) and 5 multi-objective benchmark prob- 
lems (ZDTl, 2, 3, 4 and 6). For single-objective optimization problems, a fixed 
value of rjc — 2.0 is selected as suggested in [2]. The following parameter set- 
ting is used for multi-objective benchmarks: rjc — 20.0, ri,„ — 50.0, pc — 1.0, 
p,„ = 1/ (number of variables). Each set of experiments (where 20,000 and 100,000 
fitness evaluations are conducted for single and multi-objective benchmark prob- 
lems respectively) is repeated ten times. 
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4.1 Single-Objective Functions 

n 

f{x)^Y.^l x,e [-5.12,5.12] (11) 

(=1 

Firstly, NSGAII with fixed value of rjc — 2.0 SBX operator is employed on the 30- 
variable sphere function (Eq. 11) with following parameter settings: population size 
= 100, pc = 1, Pm = 1/30, and r],n = 50. Next, SAM is embedded in NSGAll to 
dynamically update rjc- SAM can obtain better global minimal value of 0.00598655 
as compared to 0.0100824 in NSGAII with fixed ?7c- 

n 

/(x) = ^(xf + 10(l-cos(27rx,))), x,- e [-5.12, 5.12] (12) 

i=\ 

Then, we investigate the Rastrigin's function (Eq. 12) which contains many local 
optima and one global minimum (x, = 0, ; = 1, 2, 3, . . . , «). This function is diffi- 
cult to solve for global optimality using the real-coded genetic algorithm, especially 
when the initial population does not bracket the global optimum [4]. The experi- 
mental settings are the same as that of previous one for sphere function. SAM can 
obtain better global minimal value of 17.8358 as compared to 40.4383 in NSGAII 
with fixed rjc ■ 



4.2 Multi-objective Functions 

Two benchmark metrics. Inverted Generational Distance (IGD) and SPREAD are 
employed to measure the performance. IGD uses the true Pareto front as a reference 
and measure the distance of each of the solution points with respect to the front as 
Eq. 13: 



^'d} 



IGD^^ (13) 

N 

where d, is the Euclidean distance between the solution points and the closest mem- 
ber of the true Pareto front. N is the number of solution points in the true Pareto 
front. When IGD — 0, it indicates that the obtained solution set is in the true Pareto 
front. The SPREAD indicates the extent of spread among the obtained solutions and 
is computed as follows. 

df+di + y^: ~, \di — d\ 

Spread^-!—— ^'^' ' _ ' (14) 

df+di + {N- l)d 

where df and dj are the Euclidean distances between the boundary solutions (of 
the obtained solution set), di is the Euclidean distance between consecutive solution 
points. Tables 2 and 3 summarize the experimental results. 
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Table 2 Results for the inverted generational distance metric between SAM NSGA-II and NSGA- 
II 





Inverted generational distance (IGD) metric 
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Table 3 


Results for the Spread 


diversity metric between SAM NSGA-II and NSGA-II 
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3.96E-01 


2.94E-02 


ZDT6 


4.73E-01 


2.98E-02 


4.80E-01 


4.49E-02 



As observed in Tables 2 and 3, SAM achieved lower means for both IGD and 
Spread diversity metrics in all ZDT problem sets compared to NSGA-II with fixed 
distribution index. Note that no prior parameter-tuning was conducted for the runs 
using SAM. 



5 Conclusion 

Utilizing the feedback from diversity running performance metric and the crowd- 
ing distance, a self-adaptive mechanism was suggested to dynamically adjust the 
distribution index of the SBX operator. SAM is able to exploit and control the bal- 
ance between exploration and exploitation during the different evolutionary search 
stages. We demonstrated that SAM can effectively alleviate the tedious process of 
parameter tuning which is a time-consuming trial-and-error optimization process. 
On several benchmark problem sets, SAM was found to outperform NSGA-II with 
fixed distribution index. Further investigations are needed to evaluate SAM when 
applied to real-time problems where the Pareto front may be dynamic. Finally, SAM 
will also be implemented and evaluated in other MOEAs such as the Bee Colony 
Optimization and Artificial Immune System techniques. 
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A New Rearrangement Plan for Freight Cars 
in a Train 

Q-Learning for Minimizing the Movement Counts 
of Freight Cars 

Yoichi Hirashima 



Abstract In this paper, a new Q-Learning method for transfer scheduling of 
freight cars in a train is proposed. In the proposed method, the number of freight- 
movements in order to line freights in the desired order is reflected by corresponding 
evaluation value for each pair of freight-layout and removal-destination at a freight 
yard. Evaluation values are obtained by the Q-Learning method. The best transfer 
scheduling can be derived by selecting the removal- action of freight that has the best 
evaluation value at each freight-layout. 

Keywords Scheduling ■ Container transfer problem ■ Q-Learning ■ Freight train ■ 
Marshaling 



1 Introduction 

In recent years, logistics with freight train has important role in ecological aspects, 
because railway logistics is known to have smaller environmental load as compared 
to goods transportation with trucks [7]. A freight train consists of several railway 
cars, and each car has one or several containers. Commonly, goods are packed 
into containers and each container in a freight train has its own destination. Since 
freight trains can transport goods only between railway stations, modal shifts are re- 
quired for delivering them to area that has no railway. In intermodal transportations 
from the road to the rail, containers carried into the station are loaded on freight 
cars in the arriving order. The initial layout of freight cars is thus random. For 
efficient shift, the desirable layout should be determined considering destination 
of container. Then, freight cars must be rearranged before jointing to the freight 
train. 
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In general, the rearrangement process is conducted in a freight yard that con- 
sists of a main-track and several sub-tracks. Freight cars are initially placed on sub- 
tracks, then, rearranged and lined into the main track. Although similar problems are 
treated by mathematical programming and genetic algorithm [1-3, 5, 6, 8], the total 
number of movements of freight cars is not directly evaluated for realistic problems. 
Recently, a reinforcement learning method to improve marshaling plan based on the 
number of movements of freight cars has been proposed [4]. The method can obtain 
the optimal solution for simple cases by autonomous learning. 

In this paper, a new scheduling method is proposed in order to rearrange and line 
freight cars by the desirable order onto the main track. In the proposed method, the 
focus is centered on to reduce the number of car-movements that achieves desirable 
order on the main track. The optimal layout of freight cars in the main track is 
derived based on the destination of freight cars. This yields several desirable layouts 
of freight cars in the main track, and the optimal layout that can achieve the smallest 
number of car-movements is obtained by autonomous learning. Simultaneously, the 
optimal sequence of car-movements that can achieve the desired layout is obtained 
by autonomous learning. Also, the feature is considered in the learning algorithm. 
The learning algorithm is derived based on the Q-Learning [9], which is known as 
one of the well established realization algorithm of the reinforcement learning. 

In the learning algorithm, the state of sub-trucks is defined by using a layout 
of freight cars, the car to be moved, and the destination of the removed car. An 
evaluation value called Q-value is assigned to each state, and the evaluation value is 
calculated by several update rules based on the Q-Learning algorithm. Update rules 
are independent to each other and the Q-value in one update rule is referred from 
another update rule, so that Q-values are discounted according to the number of car- 
movements inside the sub-tracks. Consequently, Q-values at each state represent 
the total number of car-movements inside sub-tracks required to achieve the best 
layout from the state. Update rules of the proposed method are derived based on the 
method in [4] and modified by excluding the evaluation for car-movements between 
sub-tracks and the main track in order to improve learning performances. Moreover, 
in the proposed method, only referred Q-values are stored by using table look-up 
technique, and the table is dynamically constructed by binary tree in order to obtain 
the best solution with feasible memory space. 

In order to show effectiveness of the proposed method, computer simulations 
are conducted for two cases including a simple problem and a relatively complex 
problem. 



2 Problem Description 

The yard consist of 1 main track and m sub-tracks. Define k as the number of freight 
cars placed on the sub-tracks, and they are carried to the main track by the desirable 
order based on their destination. In the yard, a locomotive moves freight cars from 
sub-track to sub-track or from sub-track to main track. The movement of freight 
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cars from sub-track to sub-track is called removal, and the car-movement from sub- 
track to main track is called rearrangement. For simplicity, the maximum number of 
freight cars that each sub-track can have is assumed to be n, the i th car is recognized 
by an unique symbol c; (/ = 1 , . . . , ^) , and the number of sub-tracks is / . Figure 1 
shows the outline of freight yard in the case k = 30, m — n — 6. In the figure, track 
Tm denotes the main track, and other tracks A, B, C, D, E, F are sub-tracks. The 
main track is linked with sub-tracks by a joint track, which is used for moving cars 
between sub-tracks, or for moving them from a sub-track to the main track. In the 
figure, freight cars are moved from sub-tracks, and lined in the main track by the 
descending order, that is, rearrangement starts with C30 and finishes with ci. When 
the locomotive L moves a certain car, other cars locating between the locomotive 
and the car to be moved must be removed to other sub-tracks. This operation is 
called removal. Then, if A: < n • m — (n — 1) is satisfied for keeping adequate space 
to conduct removal process, every car can be rearranged to the main track. 

In each sub-track, positions of cars are defined by n rows. Every position has 
unique position number represented hy m ■ n integers. Figure 2 shows an example 
of position index for k — 30, m — n — 6 and the layout of cars for Fig. 1. 

In Fig. 2, the position "aA" that is located at row "a" in the sub-track A has the 
position number 1, and the position "fF" has the position number 36. For unified 
representation of layout of car in sub-tracks, cars are placed from the row "a" in 
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every track, and newly placed car is jointed with the adjacent freight car. In the 
figure, in order to rearrange C25, C24, C23, C22, C21, and C20 have to be removed to 
other sub-tracks. Then, since k <n -m — {n — I) is satisfied, C25 can be moved even 
when all the other cars are placed in sub-tracks. 

In the freight yard, define x, (1 < Xi < n ■ m , i — I , . . . , k) as the position number 
of the car c, , and s — [xi, ... ,Xji] as the state vector of the sub-tracks. For example, 
in Fig. 2, the state is represented by 5 = [1, 7, 13, 19, 25, 31,2, 8, 14, 20, 26, 32, 3, 
9, 15, 21, 4, 10, 5, 12, 18, 24, 30, 36, 6]. A trial of the rearrange process starts with 
the initial layout, rearranging freight cars according to the desirable layout in the 
main track, and finishes when all the cars are rearranged to the main track. 



3 Desired Layout in the Main Track 

In the main track, freight cars that have the same destination are placed at the neigh- 
boring positions. In this case, removal operations of these cars are not required at 
the destination regardless of layouts of these cars. In order to consider this feature 
in the desired layout in the main track, a group is organized by cars that have the 
same destination, and these cars can be placed at any positions in the group. Then, 
for each destination, make a corresponding group, and the order of groups lined in 
the main track is predetermined by destinations. This feature yields several desired 
layouts in the main track. 

Figure 3 depicts examples of desirable layouts of cars and the desired lay- 
out of groups in the main track. In the figured, freight cars ci,...,C6 to the 
destination) make groupj, cy, . . . , cig to the destination2 make group2, cig, . . . , C25 
to the destination3 make group3, and C26, . ■ • , C30 to the destination4 make group4. 
Groups J 2 3 4 are lined by ascending order in the main track, which make a desirable 
layout. In the figure, examples of layout in group 1 are in the dashed square. 



4 Rearrangement Process 

The rearrangement process for cars consists of following 4 operations: 

(1) selection of a freight car to be rearranged into the main track, 

(2) selection of a removal destinations of the cars on the selected car in (1), 

(3) removal of the cars to the selected sub-track, 
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(4) rearrangement of the selected car. 

These operations are repeated until one of desirable layouts is achieved in the main 
track, and a series of operations from the initial state to the desirable layout is define 
as a trial. 

In the operation (1), each group has the predetermined position in the main track. 
The car to be rearranged is defined as cj, and candidates of cj can be determined 
by excluding freight cars that have already rearranged to the main track. These can- 
didates must belong to the same group. 

Now, define r as the number of groups gi as the number of freight cars in group; 
(1 < / < r), and My, (1 < j[ < gj) as candidates of cy. 

In the operation (2), the removal destination of car located on the car to be rear- 
ranged is defined as c^ . Then, defining Uj^ (g/ + 1 < 72 < g/ + «^ — 1) as candidates 
of Cm , excluding the sub-track that has the car to be removed, and the number of 
candidates is m — 1 . 

When rearranging car that has no car to be removed on it is exist, its rearrange- 
ment precede any removals. In the case that several cars can be rearranged without 
a removal, rearrangements are repeated until all the candidates for rearrangement 
requires at least one removal. If several candidates for rearrangement require no 
removal, the order of selection is random, because any orders satisfy the desirable 
layout of groups in the main track. In this case, the arrangement of cars in sub-tracks 
obtained after rearrangements is unique, so that the movements count of cars has no 
corelation with rearrangement orders of cars that require no removal. 

Figure 4 shows an example of arrangement in sub-tracks existing candidates for 
rearranging cars that require no removal. In the figure, r — 2, where ci, C2, C3, C4 
are in groupj, C5, cg, C7, cg are in group2, and groupj must be rearranged first to the 
main track. In each group, any layouts of cars can be acceptable. In "Casel" of the 
example, the rearrangement order of cars that require no removal is ci, C2, C3, C4, 
and in "Case2", the order is C3, C2, ci, C4. Although 2 cases have different orders 
of rearrangement, the arrangements of cars in sub-tracks and the numbers of move- 
ments of cars have no difference. 



5 Learning Algorithm 

Define s(t) as the state at time t, s^ (t) — [s(t), cj] and Qi, Q2 as evaluation val- 
ues for (s(t),Uj^), (s^^ (t) , u J2) , respectively. Qi(s(t),CT) and Q2(s^(t),CM} are 
updated by following rules: 

max„ .^ Q2(s^(t + 1), Uj^) © 
(next operation is removal) , 

max„.^ Q[(s(t + r),uj^) © 
(repetitive rearrangement). 



Ql{s(t),CT) 
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(2) 



+ a[R+ y maxQi(s(t + r), Uj^(t + r))] © 
(next operation is rearrangement) 

(l-a)G2(^"^(0,Mj2(0) 

+ a[R+Kmaxe2(i^(f + l),M,/2(f + 1))] © 
(repetitive removal) 

where a is the learning rate, y is the discount factor, r is the number of direct 
rearrangements repeated between adjacent selections, and R is the reward that is 
given when one of desirable layout is achieved. 

Propagating Q-values by using Eqs. 1, 2, Q-values are discounted according to 
the number of removals of cars. In other words, by selecting the movement that has 
the largest Q-value, the number of removals can be reduced. In the learning stages. 



each Uj (l < j < gi 



P(s,Uj). 



1) is selected by the following probability: 

exp(ei(.r.»j)/T) 



E„e„^.jexp(Gi(s,")/T)' 

exp(e2(.^^«y)/T) 
Eu.uj^ exp(G2(st,")/T) ' 



(1 < ;■ < 8i) 
(gi + i<j<gi 



1), 



(3) 



where T is a thermo constant. 

The proposed learning algorithm can be summarized as follows: 

i Initialize all the Q-values as 

ii When no cars are placed on candidates of cj- , all of them are rearranged 
iii If no cars are in sub-tracks, go to ix otherwise go to iv 

iv © Determine cj among the candidates by roulette selection (probabilities are 
calculated by Eq. 3), 
® putting reward as R = 0, 
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Fig. 5 Flowchart of the learning algorithm 

© update the cotresponding Qi{s, cj) by Eq. 1, 
@ store s ' 
V If cars to be removed exists, update the corresponding Q2(s^ , cm) by Eq. 2a, 

and store (s^ , c^), otherwise update the Q(s^ , cm) by Eq. 2b and go to ii 
vi © If the car to be removed exists, remove it. The destination of the car to be 
removed is determined by roulette selection (probabilities are calculated by 
Eq. 3), 
© update the corresponding Q\is,CT) by Eq. 1, 
© store the state i'^, 

© repeat 6-©~ 6-© until all the cars on cj are removed 
vii Rearrange the cj- 
viii If there exist cars, go to ii 
ix If all the cars are rearranged, the reward R is given, and update a Q-value ac- 
cording to the last movement of car. 

Also, flowchart of the proposed learning algorithm is shown in Fig. 5. 



6 Data Structure of Look-up Table for Q-value 



In the learning algorithm explained previous section, the table lookup method is 
used for storing and referring Q-values. Since the state of the sub-tracks is repre- 
sented hy s — [x\, . . . , X]i\, (l < Xi < n ■ m, / = 1, . . . , k), the state space requires 
{nmf- memory units to store Q-values for each selection. Thus, the number of 
memory units increases by the exponential rate with increase of the total number 
of cars k. 
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Fig. 6 Structure of look-up inputi^a inputii], 

table t 

6a = 01 1001 




leaf leaf leaf leaf 
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In realistic problems the number of car is often large, so that huge memory size 
is required in order to store Q-values for all the states. Therefore, in the proposed 
method, only Q-values corresponding states that have been searched are stored. Bi- 
nary tree is constructed dynamically during the course of the learning for storing 
Q-values. 



6.1 Specification of a Q-value 



In the following, the method to specify a Q-value stored in a look-up table is ex- 
plained. The input of the table is {s, Uj), and the output is a Q-value. Assuming / 
is the order of binary description of m ■ n, the Q-value corresponding to a state s is 
specified. 

bj — bii ■ ■ ■ bii {bij = 0, 1, _/' = 1, . . . , /) is defined as the binary description of 
Xi {i = \, . . . ,k). Then, the binary description of s can be described hy b — bi-- -b^ 
of order {k + 1)7. That is, a binary tree of depth {k + 1)7 can represent s. At each 
node of the binary tree, by assigning to left descendant of the node and 1 to right 
descendant, bij can specify the descendant at the node of depth 7(( — 1) -I- /. Each 
leaf of the tree stores a state and corresponding Q-value. Given an input to the look- 
up table, the leaf corresponding to the input is specified by a search using b. When 
the input corresponds to the value stored by the leaf, the look-up table outputs the Q- 
value stored by the leaf. Otherwise, the table outputs 0. Figure 6 depicts a Q-table 
constructed by binary tree in the case ofk — m — 2, n — ?>, 7 = 3. In the figure, 
inputs ia — [1. 3], i'b — [6, 4] are given to the look-up table. Since b.j^ — [011001], 
in the former case, left, right descendants are specified from the root, the leaf stores 
the same state as the input s.^, and thus outputs Qi. While, in the latter case, since 
b\, — [110100], right, right descendants are specified, and the leaf stores s^ (^ s\,). 
That is, the state that leaf has is different from the input, and thus is output from 
the look-up table. 
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6.2 Constitution of Look-up Table 



Initially, the tree has only root that has pointer to an empty leaf. When a Q-value 
corresponding to the input state is different from initial value, the state and the Q- 
value are stored in the leaf. Then, if a new state that has the same bi\ as the stored 
state, 2 consecutive memory units for storing pointer to leafs storing data of state 
and Q-value are newly allocated. The pointer to the newly allocated memory units 
is stored in the ascendant node (in this case, root). These operations are repeated 
until bij of the new state is different from that of the stored state. If the new state 
has the different bjj, 1 memory unit is newly allocated for storing the new state and 
Q-value in addition to 2 memory units for storing pointer to leaf. The pointer to 
these 2 memory units is stored to the ascendant node. Pointers to leafs are stored in 
accordance with bjj in 2 memory units that are newly allocated. For nodes that has 
no descendant nodes or leaf, is stored as the initial value to indicate being empty. 

Whenever the state that has updated Q-value appears, the input and the state 
stored in the specified leaf are compared. When they have the same value, the stored 
Q-value is update. Otherwise, excluding bjj used to specify the stored state, 2 mem- 
ory units are newly allocated until first difference between 2 states appears in bjj, 
and then, a unit is allocated as a leaf to store 2 pairs of state and Q-value. 

Figure 7 shows an example of the look-up table for the proposed learning algo- 
rithm. In the figure, inputs s{ — [1, 3], Sg — [2, 1], s^ — [4, 1] are given to the ta- 
ble. Then, binary descriptions of s{, Sg, s^ are b{ — [i'11/712^13^21^22^23] — 00101 1, 
bfP = 010001, bg — 100001, respectively. First, sf is stored in the leaf pointed by 
the root. Next, when Sg is given, 2 memory units are newly allocated. Since i>n of 
S{ is 0, Sf and Qa are stored in the left leaf, .jf, Sg have the same bi\, and thus, 2 
memory units are allocated additionally. Since they have difference in ^112, ^f and 
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Qf are stored left leaf, and Sg and Qg are stored right leaf. Finally, when Sh is given, 
Sh and Qh are stored right leaf at the height 1 according bn — 1. 
The algorithm for look-up table construction is described below. 

i Calculate b from s and initialize i — j — I 

ii If a memory unit corresponding to bjj is a leaf then go to iii, and if it is node 
then go to iv 
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iii update /, j by 

;' ^y + 1, i ^i U <t), 

;^l, i^i + l U^I), 

and go to ii. 
iv Conduct Eq. 4 again, allocate 2 nodes for expanding a tree, and 1 leaf for storing 

state and Q-value. Then, copy data from original leaf into corresponding leaf, 

and store the pointers indicating a new leaf and nodes into ascendant nodes. 
V If bij has the same number as the state stored in the leaf, go to iv. Otherwise 

store the new input and Q-value into the corresponding leaf. 



7 Computer Simulations 

Computer simulations are conducted for 2 cases. In both cases, m — 12, n — 6, 
k = 36 and initial arrangements of cars in sub-tracks are described in Fig. 8. Also, 
desirable layout considering groups in the main track is depicted in Fig. 9. In these 
cases, the rearrangement order of groups is groupj, group2, group3, group4. Cars 
Ci , . . . , cg are in groupj , cio, . . . , cig are in group2, cig, . . . , C27 are in group3, and 
C28, • ■ • , C36 are in group4. Other parameters are set as a = 0.9, / = 0.9, R = 1.0. 

Figures 10, 1 1 show results for cases 1, 2. In the figures, horizontal axis expresses 
the number of trials and the vertical axis expresses the minimum movement counts 
of cars to achieve a desirable layout found in the past trials. Each result is averaged 
over 20 independent simulations. In both cases, the movement counts reduce as the 
number of trials increases. The optimal solution for the case 1 is simple, which can 
be obtained by removing C28, C29, C31, C32, C34, C35 into one of /-th, 10 ^ i ^ 12 
sub tracks. Then, all the cars can be rearranged directly into the main track, and 
the movement counts of cars is 42. In Fig. 10, within 200 trials, all the simulations 
derive one of the best solutions that complete a trial by 42 movements of freight 
cars. For the case 2, several removals are required for some sub-tracks to achieve 
one of the desired layouts in the main track. In Fig. 11, within 1700 trials, all the 
simulations derive the solution that completes the trial by 47 movements of freight 
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cars. In Fig. 12, one of best layouts in tlie main track obtained by the proposed 
method is shown for each case. In the figure, positions of cars in the same group 
are exchanged so that the number of removals required to achieve the layout of 
groups in the main track is reduced. Thus, in the proposed method, the layout of 
main track, the rearrangement order of cars, and the removal destination of cars are 
simultaneously optimized by the autonomous learning. 



8 Conclusions 

A new scheduling method has been proposed in order to rearrange and line cars 
in the desirable order onto the main track. The learning algorithm of the proposed 
method is derived based on the reinforcement learning, considering group of cars. 
In order to obtain the best solution with feasible memory space, in the proposed 
method, only referred Q-values are stored in look-up tables constructed dynamically 
by using binary tree. In computer simulations, by using the proposed method, the 
layout of main track, the rearrangement order of cars, and the removal destination 
of cars to achieve the optimal solution has been obtained simultaneously. 
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Coevolving Negotiation Strategies 
for P-S-Optimizing Agents 

Jeonghwan Gwak and Kwang Mong Sim 



Abstract In this paper, we consider the negotiation between two competitive 
agents that consider both time and cost criteria. Therefore, the negotiation agents 
are designed to not only optimize price utility but also be successful in optimizing 
(negotiation) speed utility. To this end, the objective of this work is to find effective 
strategies for the negotiation. The strategies are coevolved through an evolutionary 
learning process using two different evolutionary algorithms (EAs) — a genetic algo- 
rithm (GA) and an estimation of distribution algorithm (EDA). We present an empir- 
ical comparison of GA and EDA in coevolving negotiation strategies with different 
preference criteria in optimizing the price and (negotiation) speed. The experimental 
results show that both EAs are successful in finding good solutions with respect to 
both the price-optimizing (P-Optimizing) and the speed-optimizing (S-Optimizing) 
negotiation. However, both EAs are not effective in the negotiation for the con- 
current optimization of the price and speed (P-S-Optimizing negotiation). This is 
because in some cases, the original fitness function cannot characterize the differ- 
ence among P-Optimizing, S-Optimizing, and P-S-Optimizing solutions. Hence, this 
paper proposes a new fitness function that can better differentiate among the P- 
Optimizing, S-Optimizing, and P-S-Optimizing solutions. The experiments showed 
that the EAs using the proposed fitness function can coevolve effective strategies for 
the exact P-S-Optimizing negotiation. 
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1 Introduction 

Automated negotiation is a process in which a group of agents communicate with 
each other either directly or indirectly to resolve their differences in the hope of 
eventually reaching an agreement. The interacting agents negotiate to coordinate 
their activities and to reach mutually acceptable agreements about the division of 
labor and resources [1]. 

For a Grid [2] to efficiently support a variety of applications, a resource manage- 
ment system is central to its operation [3]. Grid resource management [3, 4] involves 
multiple criteria optimization, and some of these criteria are generally classified 
into time criteria and cost criteria [5]. Sim [6-9] argued and showed that negotia- 
tion agents can play an essential role in realizing the Grid vision because the agents 
can act flexibly for the Grid resources whose performance changes dynamically. 
To maintain the good performance of the system, the negotiation agents for Grid 
resource management should be designed to not only optimize price utility (i.e., 
cost criteria) but be successful in reaching early agreements (i.e., time criteria) [4, 
5]. This is because any delay in making a successful negotiation to acquire all the 
required Grid resources for Grid applications (i.e., resource consumers) before the 
deadline for executing a job will be considered as an overhead. 

Different resource owners and consumers may have different objectives, polices, 
and preferences [5, 7]. For example, consumers may have conflicting criteria be- 
tween acquiring cheaper resources and achieving faster negotiation speed (i.e., re- 
sponse time). The resource owner (respectively, a resource consumer) that prefers 
cheaper resource alternatives at the expense of having to wait longer is said to be 
more price optimizing (P-Optimizing), while a speed- Optimizing (S-Optimizing) re- 
source owner (respectively, resource consumer) prefers to obtain a resource more 
rapidly perhaps by paying a higher price at an earlier round of negotiation than its 
deadline. Different emphasis on optimizing price and optimizing negotiation speed 
can be modeled by placing different weights on the two criteria. An exact (or equally 
distributed) concurrent price and speed optimizing (P-S-Optimizing) agent has equal 
emphasis on the two criteria. In this work, we use three negotiation modes for both 
competitive agents: P-Optimizing, S-Optimizing, and PS-Optimizing negotiation. 

The idea of adopting an EDA for coevolving the negotiation strategies of the 
agents that have different preference criteria such as optimizing price and optimiz- 
ing negotiation speed was first proposed in [5]. The problem of ineffectiveness in 
coevolving negotiation strategies for the P -S-Optimizing negotiation is presented in 
[17]. Furthermore, in [17], one possible solution for resolving the difficulties of a P- 
S-Optimizing negotiation is suggested by restricting the solution space (which is the 
EA's perspective; from the agents' perspective, the negotiation corresponds to the 
adoption of a feasible strategy space) using predefined strategy profiles. However, 
in some cases, the fitness function originally used in [5] and [17] cannot effectively 
discriminate between the different emphases on the two criteria in the total utility 
space. To solve this problem, we propose a new fitness function that can better char- 
acterize the differences among P-Optimizing, S-Optimizing, and P-S-Optimizing so- 
lutions. 
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The rest of this paper is organized as follows: Section 2 specifies the negoti- 
ation model of this work. The coevolutionary framework based on GA and EDA 
is described in Sect. 3. The problem of coevolving strategies for P-^-Optimizing 
negotiation and its solution will be presented in Sect. 4. Section 5 presents the ex- 
perimental results and analyses. Finally, Sect. 6 concludes this work with a summary 
and suggestions for future work. 



2 P-S-Optimizing Negotiation 

In a classical bargaining model, the utility function C/^ of agent x, where x e {B, S] 
and X denotes x's opponent, is defined as follows: Let IPx and RP^ be the initial 
and the reserve prices of x. Let D be the event in which x fails to reach an agree- 
ment with its opponent x. U" : [IPx,RPx] U Z) ^ [0, 1] such that C/^(Z)) = and 
U^{Px) > U^{D) for any possible proposal Pj e [IPx,RPx]- If -f and x are sensi- 
tive to time, let tx be the deadline of x and tj be the deadline of x. An agreement 
price that is acceptable to both B and S lies within the interval [RPs, RPb]- 

In the bargaining model with complete information between B and S, both agents 
know the opponent's initial price, reserve price, and deadline. If one of the agents 
has a significantly longer deadline than its opponent, the agent that has a longer 
deadline will have sufficient bargaining advantages. In other words, an agent that has 
the longer deadline will dominate the negotiation. Under these conditions, Sim [10, 
11] proved that an agent's optimal strategy can be computed using its opponent's 
deadline and reserve price. It can be stated as the following theorem: 

Theorem 1 If agent x 's deadline tx is significantly longer than its opponent's dead- 
line x^ {i.e., tx ^ Tf ), agent x achieves its maximal utility when it adopts the strat- 
egy kx ^log^ii^f^). 



2.1 Utility Functions 

The total (aggregated) utility function C/^ of agent x is composed of two attributes — 
price and time (i.e., the number of negotiation rounds) — and is defined as follows. 

U'iPx, Tx) ^wpx U'p{Px) + WSX Ul(Tx) (1) 

where Px e {0, Pq} and Tx e {0, 7c}, and Pc and Tc is are the price and time 
at which an agreement is reached. Up(Pc) e [0, 1] is the price utility of x, and 
U'liTc) e [0, 1] is the speed utility of x. wp and ws are the weighting factors for 
price and (negotiation) speed, respectively, and wp -\- ws — l. 
The price and speed utilities are given as follows: 
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«min + (1 - "min) (^p7=7F;) ' if ^n agreement is reached 



(3) 



0, otherwise 

"min "I" *-i ~ "minH^ ~ r'')' if an agreement is reached 
0, otherwise 

where u^^^ is the minimum utility when x receives a deal at its reserve price and 
"min i^ ^^ minimum utility when x receives a deal at its deadline. For the experi- 
mental purpose, the values of u^^^^ and u^^^ are defined as 0.1. 

If X does not reach an agreement before its deadline, Up — U^ — 0, and thus, 
C/^ = 0. Otherwise, U'iPc, Tc) > 0. 



2.2 Negotiation Strategies 

This work considers the bilateral negotiation between B and 5 with incomplete 
information in which both agents do not know each other's deadline and reserve 
price. Both B and S are sensitive to time; further, we adopt the time-dependent 
strategies proposed in [12]. The proposal /"/ of jc at time f , < f < r^ is defined as 
follows: 

P^ = IP^ + i-ir (^) ' \RPx - IPx\ (4) 

where a = for Z? and a — \ for 5', and < Xjr < oo. 

The time-dependent strategies can be classified into three categories: (i) con- 
servative (conceding slowly, Xx > 1), (ii) linear (conceding linearly, X^ — I), and 
(iii) conciliatory (conceding rapidly, < X^ < 1) [12, 13]. 



2.3 Negotiation Protocol 

The negotiation between B and S is carried out using Rubinstein's alternating offers 
protocol [14]. B and S can conduct the negotiation only at discrete time points (e.g., 
in this work, S makes an offer at f = 0, 2, 4, 6, ... , and B makes a counter-offer at 
f = 1, 3, 5, 7, . . .). The negotiation process ends in both cases: (i) once an offer or 
a counter-offer is immediately accepted (i.e., an agreement is reached) by the other 
one or (ii) once the earlier deadline is reached without any agreement. In the latter 
case, the negotiation process ends in a conflict, and the utility outcome will be zero. 



2.4 Objective 

For the given different deadlines and different preferences of the cost and time cri- 
teria (i.e., different values of wp and ws), agents will face different opponents with 
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different deadlines and strategies. Under these conditions, the objective of this work 
is to find an effective strategy Xx that would optimize ViPx, Tx). In this work, 
(evolutionary) learning is based on two asymmetric populations in which each pop- 
ulation has its own fitness evaluation. Agents learn effective strategies by interacting 
with individuals from the other population through random pairing. In the following 
section, the detailed procedure will be described. 



3 Coevolutionary Models for P-S-Optimizing Negotiation 

When populations between two or more species interact, each may evolve in re- 
sponse to the characteristics of the other. The natural coevolution refers to the mu- 
tual (or inter-dependent) evolution between interacting populations. The survival 
skills of the natural coevolution by making mutually beneficial arrangements have 
long inspired scientists to develop coevolutionary algorithms for highly dependent 
problems in which there are strong interactions between two elements or among 
several elements. 

In the bilateral negotiation problem domain, inter-population based coevolution 
having two populations is considered. The fitness of each individual of one popula- 
tion depends on each individual of the other population, and hence, an individual's 
fitness landscape is not fixed but coupled. Therefore, coevolution is regarded as a 
type of landscape coupling where adaptive moves by one individual can potentially 
change the landscape of the other. The interaction between two populations comes 
from the pairing of strategies of each population, and therefore, a successful pairing 
mechanism is important. Furthermore, to achieve a better performance, the resulting 
pairing should make a sufficiently prevailing set in the feasible set. In this work, we 
use one-to-one random paring because of its simplicity and efficiency. 

To coevolve effective strategies under different deadlines and different weights 
of price and speed preferences, a coevolutionary framework using real-coded GA 
and EDA is implemented. B and S have each of their populations d^ and d^ consist 
of a set of candidate strategies. D^ and D^ are the mating pool (MP) of B and 5', 
respectively. 

The evolution of the strategies of one population affects the strategies of the 
other population. In the process of coevolving strategies, each individual of the two 
populations will negotiate with the other through one-to-one random pairing. The 
fitness value of each individual is determined by its negotiation outcome. The details 
of the GA and EDA are as follows: 



3.1 Encoding Scheme 

A binary coding mechanism has drawbacks because of the existence of Hamming 
cliffs and the lack of computation precision [15, 16]. Therefore, in the GA and 
EDA, each negotiation strategy of the populations is encoded as a real number. Each 
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Table 1 Characterizations of GA and EDA 
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EA type 


Method 


Details 


GA [17] 


Selection 


^-tournament selection 




Crossover 


Heuristic crossover 




Mutation 


Gaussian mutation 


EDA: 
UMDA [18] 


Selection 

Estimation of 
probabilistic model 


Truncation selection 

^ \J Ino 

where X = (A,i, A2, ■ ■ ■ .^w}, 

M = ff Efcl ^M and a = y 5^ Y.tx(h - M)^ 




Sampling individuals 
from the model 


Sampling normal random variables 
(individuals) from //(X) [18, 19] 


GAand 
EDA 


Stopping criteria 


Terminate when either 

(i) g > G™^" or (ii) |/i;„, - f^\.^\ < &'"'P 



individual in a population is mainly represented by the strategy that it has. For the 
experimental purpose, we consider the range of strategies for Xb and Xs in [0, 10]. 



3.2 Fitness Function 

To evaluate each individual of a population, the fitness function f{x) is defined as 
follows: 



fix) = U-HP„ T,) ^wpx Uf(P,) + WSX U's(T,) 



(5) 



In each generation g, randomly pick one individual from D^ and randomly pick 
the corresponding individual from Z)? . Each selection procedure is carried out with- 
out a replacement. Then, the selected individuals of one population will negotiate 
with the selected individuals of the other population in a one-to-one manner. The 
values of the fitness function will be computed using the resulting negotiation out- 
comes. Finally, if an agent x reaches an agreement with its opponent. Up (7\ ) > 0, 
Ug{Tx) > 0, and f(x) — U^(Pj(, T^) > 0. If a negotiation is terminated without an 
agreement, f(x) — U^(Px, Tx) — 0. 



3.3 EAsfor Coevolution 



The characterizations of GA and EDA used in this work are described in Table 1. 
For more detailed information of the GA and EDA, refer to [17]. 

The coevolution procedure for finding the two types of effective strategies (i.e., 
Xb and Xs) from the two populations is described in Algorithm 1. The interaction 
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between the two populations is in the fitness evaluation stage (at lines 3 and 8 in 
Algorithm 1). 

Algorithm 1 EAs (GA and EDA) for coevolving negotiation strategies 
1: set the search space (Imin: ■^max) and the generation counter g -^0 
2: generate D^ of M randomly generated individuals 
3: evaluate individuals in D^ using the fitness function 
4: WHILE the stopping criteria are not satisfied 
5: select A^(< M) individuals d'^ from D^ using the corresponding 

selection method 
6: compute the average fitness values of individuals in d^ 

7: GA: produce N individuals using the corresponding 

reproduction mechanism; 

the MP consists of the produced A'^ individuals and d^ 
EDA: produce M individuals using the corresponding 
reproduction mechanism; 
the MP consists of the produced M individuals 
8: evaluate individuals in D^ using the fitness function 

9: g^g+l 

10: ENDWHILE 
1 1 : extract the individual with the highest gene value in high fitness value region 



4 Problem of P-S-Optimizing Negotiation and Its Solution 
4.1 Problem of P-S-Optimizing Negotiation 

The GA and EDA described in the previous section cannot coevolve the appro- 
priate strategies for P-S-Optimizing negotiation because the original fitness func- 
tion (which is the same as the utility function described in Sect. 3.2) cannot char- 
acterize different emphases (i.e., preference levels) on the cost and time criteria 
for some cases. That is, for the given values of uip and ws, the utility func- 
tion U^{Px, Tx) given in Sect. 3.2 cannot appropriately characterize the difference 
between Up{Px) and Uj(Tx) in some cases. For example, consider the case of 
the exact P-S-Optimizing negotiation setting (wp, ws) — (0.5, 0.5). The total util- 
ity U^Px, Tx) of the set U^iP^) = 0.7 and [/^(rj = 0.3 (which corresponds 
to a more P-Optimizing solution) will have the same value for the total utility 
ViPx, Tjc) of the set t/^(P,) = 0.3 and U^{T,) = 0.7 (which corresponds to a 
more S-Optimizing solution). Furthermore, it is the same for the set U'p{Px) — 0.5 
and Uj(Tx) — 0.5 (which corresponds to the exact P-S-Optimizing solution). The 
above example shows that in some cases, the original fitness function cannot char- 
acterize the difference between P-Optimizing, S-Optimizing, and P-S-Optimizing 
solutions. 
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4.2 Proposed Fitness Function 



The ambiguity of tiie original fitness function in tiie total utility space is solved 
by adopting the proportion of weighting factors (which represent the preference 
levels in the total utility space) in the calculation of the proposed fitness function. 
The key idea of the proposed utility function is to directly measure the difference 
(or similarity) between (i) the ratio of price and time weighting factors, and (ii) the 
corresponding ratio of the price and time utility functions. As the difference between 
them decreases, the value of the fitness function will be considerably high. The 
proposed fitness function is designed as follows: 



(Mode 1) for P -Optimizing: f(x) — 1 



(Mode 2) for S-Optimizing: f(x)—l 



ws 

Wp 

wp 
ws 



UUTc) 



Uf.iPc) 
U'piPc) 



(Mode 3) for P-S-Optimizing: 



./W=l 



ws_ 
Wp 



UUTc) 



UUPc) 



Wp 
ws 



Us(Tc) 



UUPc) 



UUTc) 



5 Empirical Evaluation 

A series of experiments was carried out to evaluate the performance of the original 
fitness function (in Sect. 3.2) and the proposed fitness function (in Sect. 4.2). 



5.1 Experimental Settings 

An empirical comparison of GA and EDA is also presented to determine which 
model is more suitable for coevolving negotiation strategies with different prefer- 
ence criteria for optimizing price and speed. The experimental parameter settings 
are as in Table 2. 

For the purpose of the experiments, both competitive agents have the same 
weights of the preference criteria for the three negotiation modes. The settings 
for each negotiation mode are as follows: (wp, ws) — (1.0, 0.0) for P-Optimizing 
negotiation, (wp, ws) — (0.1,0.9) for S-Optimizing negotiation, and {wp, ws) — 
(0.5, 0.5) for P-S-Optimizing negotiation. 



5.2 Optimal Conditions 



In the case of the P-Optimizing negotiation, we will experimentally prove the 
properness by examining two extreme cases: Case I and Case II. 
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Table 2 Parameter settings for EAs and negotiation 

Parameters Values 

GA and EDA 



Population size (^V) 






25 


Mating pool size (M) 






50 


Maximum number of g 


enerations (G""'^) 


2500 


Threshold for stopping 


criteria 


(S""P) 


10-4 


Tournament size (k) 






3 


Crossover rate (Pcx) 






0.7 


Mutation rate (Pmu) 






0.002 


{1Pb,RPb) 






(5, 85) 


UPs,RPs) 






(95, 15) 


(^mini ^max) 






(0, 10) 


Deadline: Long 






100 


Deadline: Mid 






50 



Negotiation 



Case I If an agent B has a sufficient bargaining advantage over S(i.e., xb ^ ts), B 
will dominate the negotiation irrespective of whether 5* adopts any strategies. The 
optimal negotiation strategy of B will be determined by Theorem 1 . For example, 
under the negotiation parameter settings described in Table 1, the optimal strategy 
of B is when Xb — 3, and the agreement is reached at Pc — 15 and Tc — 50. The 
strategy of S and Xs will not converge to a specific value, and thus, it will have 
a dynamic range of values. We describe this dynamic range as [Min. value. Max. 
value]. 

Case II When both B and S do not have any bargaining advantage (i.e., rg — xs), 
we can think that the agreement of the negotiation with the abovementioned ne- 
gotiation parameters should be reached at a Pareto optimal point (i.e., Pc — 50). 
However, in a practical negotiation model, the point does not always follow the 
Pareto optimal point since the agent that proposes the first negotiation proposal (in 
our experiment S does) usually gets a lower payoff (or utility) in the alternating of- 
fers protocol. Under the Pareto optimal condition, the optimum strategies are when 
both Xb and Xs equal X^ax — 10 (i.e., both B and S do not concede at all). 

In the case of S-Optimizing and P-S-Optimizing negotiations, there is no such 
theory as that in the case of P -Optimizing negotiation to prove the optimality of 
the solutions. However, the evaluation can be carried out by examining whether the 
solutions follow the general characteristics for the given negotiation mode. For ex- 
ample, in the case of the S-Optimizing negotiation, the agreement should be reached 
at an earlier negotiation time (than its deadline) by paying a relatively high price. 
The abovementioned two extreme cases are also used for evaluating the optimality 
of the solutions. Interestingly, according to Proposition 7 given in [13], irrespective 
of the deadline, agents with linear strategies are more likely to make deals than those 
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with conservative strategies while achieving higher utiUty than those with concilia- 
tory strategies. Hence, linear strategies (i.e., Xg — 1-0 and Xs — 1.0) are optimal 
solutions for the exact P-S-optimizing negotiation. 



5.3 Experimental Results 

The results of these two extreme cases for the original utility function and the pro- 
posed utility function are shown in Tables 3-6. 



5.3.1 Results Obtained Using Original Fitness Function 

The results are listed in Tables 3 and 4. The original fitness function in Eq. (5) is the 
same as the total utility function in Eq. (1). 

Observation 1 Both GA and EDA with the fitness function in Eq. (5) find good 
strategies for P -Optimizing and S-Optimizing negotiation, respectively. 

Analysis In both Case 1 (in Table 3) and Case II (in Table 4), Xg and Xg has close 
to optimum solutions, as discussed in Sect. 5.1. 

Observation 2 Both GA and EDA with the fitness function in Eq. (5) cannot find 
effective candidate solutions for the P-S-Optimizing negotiation. 

Analysis In both Case I (in Table 3) and Case II (in Table 4), Xb and Xs did not 
converge to the values that we expected (Similarly, by using the EDA proposed in 
[5], we found that Xb and Xs did not converge in the case of PS-Optimization). 
This is because more conceding strategies will be cut off as the evolution proceeds 
since these strategies will get a lower payoff than the other strategies (i.e., linear or 
conservative strategies) eventually. Hence, Xg and Xs of both GA and EDA tend to 
be more P- Optimizing. 



5.3.2 Results Obtained Using Proposed Fitness Function 

The results are listed in Tables 5 and 6. The proposed fitness function does not equal 
the utility function any longer. The proposed fitness function measures the similar- 
ity between the ratio of weighting factors and the ratio of price and speed utili- 
ties. For example, for the P-Optimizing negotiation (i.e., (wp, ws) — (1.0,0.0)), 
and in Case I (i.e., B has significant bargaining advantage than S), the opti- 
mum utilities are obtained at (Pc, Tc) = (15, 50). For B, Uf{Pc) = 0.8875 and 
Ug{Tc) — 0.55. Hence, the optimum value of the proposed fitness function is 
fB ( -._i_\ws_ _ Us(Tc) I _ J _ lao _ 0.55 I -0 3803 For 5 U^(Pr)-0 1 
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and U^(Tc) — 0.1. Hence, the optimum value of the proposed fitness function is 
f^p,{x) = 1 - I ff - ^1^ I = 1 - I ^ - 1^ I = 0.0. Likewise, for different prefer- 
ence criteria and negotiation parameter settings, different optimum (i.e., maximum) 
values of the fitness function will be drawn from the proposed fitness function (e.g., 
for the above example, the maximum value /™,(x) for B is 0.3803). More analyses 
and experiments related to this issue will be presented in our future paper. 

Observation 3 Both GA and EDA with the proposed fitness function find effective 
strategies for the P-Optimizing negotiation. 

Analysis The results show that both GA and EDA using the new fitness function 
achieved good performance. In Case 1 (in Table 5), Xb converges close to the values 
of the optimum solutions. Furthermore, in Case 11 (in Table 6), Xb and Xs converge 
close to the values of the optimum solutions. 

Observation 4 In Case 1, the GA with the proposed fitness function finds effective 
candidate solutions for the S-Optimizing negotiation. However, the EDA cannot co- 
evolve S-Optimizing strategies. In Case II, both GA and EDA can coevolve effective 
S-Optimizing strategies. 

Analysis In Case 1 (in Table 5), the GA can get more conceding strategies that can 
reach early agreements. However, the EDA cannot find solutions at all. This result 
shows that the EDA does not have a sufficient search capability to find solutions 
for the S-Optimizing negotiation. In Case II (in Table 6), both GA and EDA can 
find effective strategies that are considerably more conceding strategies than linear 
strategies (i.e., Xb = 0.1561 and Xb = 0.0026 for the GA, and Xb = 0.2595 and 
Xb = 0.0090 for the EDA). 

Observation 5 In Case I, both GA and EDA with the proposed fitness function 
find effective candidate solutions for the P-S-Optimizing negotiation. The EDA has 
better performance than the GA in terms of the evolution speed. In Case II, both GA 
and EDA can coevolve effective P-S-Optimizing strategies. 

Analysis In Case I (in Table 5), the GA needs considerably more generations to 
converge to solutions than the EDA (i.e., 2059.01 generations for the GA, and 59.03 
generations for the EDA). This result shows that the EDA has better performance 
with respect to coevolving effective P-S-Optimizing strategies in terms of the evolu- 
tion speed (i.e., the number of generations required for the coevolution). In Case II 
(in Table 6), both GA and EDA can coevolve effective P-S-Optimizing strategies 
that are close to the value of the linear strategy (i.e., Xb — 0.9519 and Xb — \ for 
the GA, and Xb = 0.9508 and Xb = 1 for the EDA). 
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6 Conclusions and Future Work 

This paper provides empirical evidence for coevolving negotiation strategies for the 
negotiation between the two competitive agents having different preference criteria 
for optimizing price and negotiation speed. Two rather different evolutionary ap- 
proaches, EDA and GA, were respectively used and compared for coevolving the 
strategies. The experimental results (given in Tables 3 and 4) showed that adopt- 
ing the GA and EDA were a good choice for coevolving effective strategies for the 
P-Optimizing and the S-Optimizing negotiations. However, we found that both GA 
and EDA did not converge to proper solutions (that we expected in Sect. 5.1) in 
the case of the P-S-Optimizing negotiation. The main problem of the failure in co- 
evolving the strategies was that the original fitness function (given in [5] and [17]) 
could not effectively characterize the different weights on the cost and time pref- 
erence criteria in the total utility space. On the basis of the analysis, we proposed 
the new utility function in Sect. 4.2. The experimental results (given in Tables 5 
and 6) showed that the proposed method achieved more reliable results in the case 
of the P-S-Optimizing negotiation than the method used in [5] and [17]. However, 
the results also showed that the EDA using the new fitness function did not coevolve 
effective strategies for the S-Optimizing negotiation, and the GA using the new fit- 
ness function was not effective in the case of the P-S-Optimizing negotiation since it 
required considerably more generations than EDA. To develop more reliable models 
that are successful and efficient in all cases, a hybrid model [20] can be adopted to 
compensate for the defects of each evolutionary approach by combining the GA and 
the EDA. 

Our future work includes an exhaustive analysis that considers (i) more combi- 
nations of different preference weights between the cost and the time criteria (e.g., 
more P-Optimizing such as (wp, ws) — (0.7, 0.3) and more S-Optimizing such as 
{wp, ws) — (0.3, 0.7) cases) and (ii) heterogeneous negotiation (e.g., the case when 
one agent is P-Optimizing, while the other agent is S-Optimizing, and all these types 
of cases). Furthermore, by reducing and focusing on the feasible solution space, the 
restriction scheme of the solution space in [17] can help to reduce the computational 
overhead and to obtain more high-quality solutions. 

Acknowledgement This work was supported by the Korea Research Foundation Grant funded 
by the Korean Government (MEST) (KRF-2009-220-D00092). 
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Policy Gradient Approach for Learning 
of Soccer Player Agents 

Pass Selection of Midfielders 



Harukazu Igarashi, Hitoshi Fukuoka, 
and Seiji Ishihara 



Abstract This research develops a learning method for the pass selection problem 
of midfielders in RoboCup Soccer Simulation games. A policy gradient method is 
applied as a learning method to solve this problem because it can easily represent 
the various heuristics of pass selection in a policy function. We implement the learn- 
ing function in the midfielders' programs of a well-known team, UvA Trilearn Base 
2003. Experimental results show that our method effectively achieves clever pass 
selection by midfielders in full games. Moreover, in this method's framework, drib- 
bling is learned as a pass technique, in essence to and from the passer itself. It is 
also shown that the improvement in pass selection by our learning helps to make a 
team much stronger. 

Keywords Multi-agent system • Pass selection ■ Policy gradient method ■ 
Reinforcement learning ■ RoboCup 



1 Introduction 

Recently, much work has been done on the learning of coordination in multi-agent 
systems [1, 2]. The RoboCup Simulation League 2D is recognized as a test bed 
for such research because there is no need to control real robots and one can 
focus on learning coordinative behaviors among players. However, multi-agent 
learning continues to suffer from several difficult problems such as state-space 
explosion, concurrent learning [3], incomplete perception [4], and credit assign- 
ment [2]. These four problems should be studied and resolved to make multi- 
agent learning successful in the games of the RoboCup Simulation League 2D 
(Fig. 1). 
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Fig. 1 RoboCup Soccer 
Simulation League 2D y 




As an example of multi-agent learning in a soccer game, Igarashi et al. proposed 
and applied a reinforcement-learning approach to realizing coordination play be- 
tween a kicker and a receiver in direct free kicks [5]. They dealt with a learning 
problem between a kicker and a receiver when a direct free kick is awarded just 
outside the opponent's penalty area. In such a situation, to which point should the 
kicker kick the ball? They proposed a function that expresses heuristics to evaluate a 
candidate target point for effectively sending/receiving a pass and scoring. However, 
they dealt only with the attacking problems of 2v2 (two attackers and two defend- 
ers). In this chapter, we introduce our work in which we applied the method to a 
pass selection problem of four midfielders in a full soccer game [6]. 



2 Coordination of Soccer Agents 

2.1 Cooperative Play in RoboCup Soccer Simulation 

Reinforcement learning is widely used [7, 8] in the research areas of multi-agent 
learning. In the RoboCup Soccer Simulation League, Andou used Kimura's stochas- 
tic gradient ascent (SGA) method to learn the dynamic home positions of 1 1 players 
[9]. Riedmiller et al. applied TD learning to learn such individual skills as intercept- 
ing the ball, going to a certain position, or kicking and to select those individual 
skills [10]. They investigated attacking problems with 2vl, 2v2, 3v4, and 7v8. Stone 
et al. studied keepaway problems [11] with 3v2 and half-field offense problems [12] 
with 4v5 using Sarsa [7] to learn the selection of macro behaviors such as ball hold- 
ing, passing, dribbling, and shooting. 



2.2 Coordination at Free Kicks 

In the previous section, we cited several research efforts on the cooperative behav- 
iors of soccer agents. However, a crucial problem remains. In the previous research. 
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each agent apparently learns its policy of action selection "autonomously" to com- 
plete the given task. Riedmiller et al. assumed that all agents share input informa- 
tion, i.e., the x-y positions of all players and the ball, with other agents [10]. Stone 
et al. used other agents' experiences, which are time-series data on state, action, 
and reward, to accelerate learning in a large problem. For that purpose, agents must 
communicate their experiences with their partners to facilitate sharing information 
among them. If agents share input information and experiences with other agents, all 
agents will obtain the same value function by learning. This will simplify the real- 
ization of various cooperative plays among agents. However, if agents' observations 
are imperfect or uncertain, none of the agents can share the same input information 
with all other agents. Without perfect communication among agents, they cannot 
share their experiences with each other. Moreover, if only agents that have identical 
value functions are assumed, agent individuality and division of roles among them 
will not emerge from learning. 

Igarashi et al. proposed a method where all agents learn autonomously without 
assuming perfect communication or identical input information [5]. They applied it 
to an attacking problem with 2v2 when a direct free kick is awarded just outside the 
opponent's penalty area. We briefly summarize the method in the next section. 



3 Learning by a Policy Gradient Method 
3.1 Policy Gradient Method 

A policy gradient method is a kind of reinforcement learning scheme that originated 
from Williams's REINFORCE algorithm [13]. The method locally increases and 
maximizes the expected reward per episode by calculating the derivatives of the ex- 
pected reward function of the parameters included in a stochastic policy function. 
This method, which has a firm mathematical basis, is easily applied to many learn- 
ing problems. One can use it for learning problems even in non-Markov Decision 
Processes [14, 15]. It was applied to pursuit problems where the policy function con- 
sists of state-action rules with weight coefficients that are parameters to be learned 
[15]. The definition of non-Markov Decision Processes is shown in Appendix 2. 



3.2 Stochastic Policy for Action Decision 

Igarashi et al. [5] state policy 7T{a; s, co) for determining action a(e A) of all agents 
when a multi-agent system is in state .?(e S), and this policy is given stochastically 
by a Boltzmann distribution function with object function E(a\ s, u>) as 

n{a\ s, cti) = exp(— £(0!; s, ft))/r) / 2_, exp(— ^(x; s, &))). (1) 

xeA 
Weight parameters ai — {a>j} (j — 1,2,..., A^^,) in (1) are determined by a policy 
gradient method described in the next section. T is a parameter called temperature. 
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3.3 Autonomous Action Decision and Learning 



For the autonomous action decisions and the learning of each agent, policy function 
7T(a; s,a>) for the entire multi-agent system was approximated by the product of 
each agent's policy function [14-16] n^iax; sx, {«)}) as 



Tt{a\s,(j))^Y[^i^{'^^'^'^i^^ {'")})' 



(2) 



where ax is the action of agent X and sx is the state perceived by agent A. oi^ is the 
y'-th parameter in agent A.'s policy function itxiflx; sx, {w^:})- In (2), it seems that the 
correlation among agent action decisions is neglected. However, each agent can see 
other agents' states, even if they are not perfect, and use them in its policy function 
TTxiax', s, {cii^}). Thus, the approximation in (2) will partially contribute to learning 
of coordination among agents. 

Weassumethatagent A. 's policy function 7r;t(ax; s, Ico^}) is also given by aBoltz- 
mann distribution function with objective function Exiax; sx, {«)}) defined by 



Ex{ax\sx, {«}}) ^-'Yl,'^) ■ Uj{ax;sx), 



(3) 



where function U j{ax\ sx) is the y-th heuristics that evaluates action ax [5]. 

At the end of each episode, common reward r is given to all agents that made 
their decisions during the episode. The derivative of expectation of reward E[r'\ for 
parameter w^ is shown as 



dE[r]/da))^E 



L-\ 



Evw 



f=0 



(4) 



where L is an episode length. 

With (2), characteristic eligibility go, on the right-hand side of (4) can be written 
as [14, 15] 

e„A(0 = 91n7r(fl(0; ^(0, {«)})/9«) ^ d\mix{axit)- sxit), {«)})/9«). (5) 



Substituting (2) and (3) into (5), the policy gradient in (4) gives a learning rule as 

L-l 



Aft;) = £r^ Uj{ax{t))-Y,Uj{ax)Trx{ax\sx{t),[w)]) IT 

1=0^ ax -I' 



(6) 



where ax{t) is an action actually selected by policy ttx and sxit) is a state perceived 
by agent X at time t. Each agent updates w^: by the learning rule in (6) at the end of 
each episode [5]. 
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4 Pass Selection Problem 

4.1 Pass Selection in Soccer Games 

A pass is a typical cooperative play between two players in soccer games. Determin- 
ing the receiver from among teammates is very important in a pass play. The first 
action to consider is passing the ball to a receiver safely. However, useless iteration 
of backward passes should be avoided. Thus, a player must select a receiver who 
stands in a position to receive the pass safely and who has a relatively high possi- 
bility of scoring a goal after receiving the ball. For this purpose, we use heuristics 
functions that seem to be useful for selecting a pass receiver as Uj{ax; sx) in (3), 
where ax is a passer I's action of selecting a pass receiver and passing the ball to 
the receiver. 



4.2 Reward 

In this paper, we apply the policy gradient method summarized in Sect. 3 to pass 
selection problems of midfielders. Dribbling is considered a pass from and to the 
passer itself. If midfielders can pass and dribble the ball safely without being inter- 
cepted, the length of time their team holds the ball will be longer. Keeping the ball 
for as long a time as possible is obviously one of the effective strategies of mid- 
fielders in a soccer game. Here, we define the term "team X is keeping the ball" as 
the situation where the nearest player to the ball is a member of team X and this 
situation continues for a duration of more than five simulation cycles. 

We define learning episode a of team X by a history of states and actions 
of agents while team X keeps the ball, and we define reward r of episode a as 
r((7) = — 1/L(ct). Length L(a) of episode ct is defined by the difference between 
the time when team X begins to keep the ball and the time when the ball is taken 
by the opponent team. Since r takes a negative real value, it is actually a penalty 
rather than a reward. The shorter the duration time of keeping the ball is, a larger 
common penalty r(a) is given to team X's midfielders who made pass selection 
during episode a . 



4.3 Heuristics for Pass Selection Problems 

We used five heuristics from U\ to f/5 for evaluating the current position of a team- 
mate as a desirable pass receiver, [/j , U2 and C/3 are heuristics for passing the ball 
safely. Ua, is heuristics for making an aggressive pass. C/5 is used for treating reliable 
information as more important knowledge than unreliable information. All [/, are 
normalized between and 10. 
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The meanings of the five heuristics are summarized as follows. Ui considers the 
existence of opponents on the pass course. U2 evaluates a distance between a re- 
ceiver and the opponent nearest to the receiver, t/3 takes into account the number 
of opponent players around a receiver. U4 expresses a heuristics that the nearer re- 
ceiver to the opponent's goal has a greater chance of scoring a goal. C/5 evaluates the 
degree of certainty of a receiver's position perceived by the passer. The definitions 
of the five heuristics are shown in Appendix 1 . 



5 Experiments 

5.1 Team Used in Learning Experiments 

We used UvA Trilearn Base 2003 as a base team for learning experiments. UvA 
Trilearn 2003 is a team of the University of Amsterdam and champion over 46 
qualified teams in the 2D Simulation Soccer League of RoboCup2003 Padova. The 
team released a part of the source code of UvA Trilearn 2003 [17]. 

According to the home page of the UvA team [17], the released code contains 
low-level and intermediate-level implementation (agent-environment synchroniza- 
tion method, world model, player skills) but not a high-level decision procedure. 
Instead, they have included a simple high-level action selection strategy. The fastest 
player to the ball intercepts the ball and shoots it to a random corner in the oppo- 
nent's goal regardless of his position on the field. The remaining players move to 
a strategic position determined by their home position in the formation and by the 
position of the ball. We modified the code slightly and implemented the learning 
function defined in Sects. 3 and 4 for UvA's four midfielders. We call this team 
"UvA Trilearn Base 2003," or "UvA Base" for short, and use it in our learning 
experiments. 



5.2 Learning Experiments 

UvA Base with four midfielders who learn pass selection plays 50 games against 
UvA Base with midfielders who randomly make passes to teammates. Initial values 
of &)^ in (3) are set to 0. T and e in (6) are set to 10.0 and 0.1, respectively. 

The change in rewards r given to four midfielders is shown in Fig. 2, where 
<r>5o is an average over the last 50 episodes. Midfielders have their own set of Wy 
and learn their values by playing 50 games. The results of co^ are nearly identical 
to each other. Therefore, only the results of a midfielder called MF_8 are shown in 
Fig. 3. 

Reward averages <r>5o for all midfielders do not increase in Fig. 2, even as 
the learning proceeds. At a glance, the learning seems ineffective and a waste of 
time. However, the effectiveness of learning is shown by evaluation experiments 



Policy Gradient Approacli for Learning of Soccer Player Agents 



143 



Fig. 2 Reward average over 
the last 50 episodes 



MF 6 -^MF 7 



MF 8 



MP 9 



16 21 

Episode{lunit=5Q) 




■Q.OB 

Reward average </>j(i 



Fig. 3 Change in Wj of 
player MF_8 




101 201 301 401 501 601 701 801 901 1001 1101 

Episode 



in the next section. The phenomenon of reward averages not increasing is due to 
the following reason. A player in UvA Base is programmed to keep a close watch 
over the opponent goal when it possesses the ball. This increases confidence in the 
information of a teammate's position in the forward direction of a passer, and it 
increases the importance of a forward pass in objective function E(ax; s, {iw^}) in 
(3). Thus, a large value of w^ in Fig. 3 causes an agent to make a more offensive pass 
and to play in a deeper area of the opponent team's territory as learning proceeds. 
Playing in a deep area of the opponent prevents the duration of keeping the ball from 
getting longer. 

The ratio among ai^ seems to converge in Fig. 3. This means that there is a 
strong attractor point in the parameter space of co^ and that our learning captures 
this attractor point. Increasing iw^ with a fixed ratio in magnitude is equivalent to 
annealing in a stochastic policy in (1). Accordingly, a deterministic pass selection 
algorithm based on m^ at the attractor point was obtained as policy Jtxiax; 5, {&)^}) 
by the learning experiments in this section. 
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Table 1 Results of 30 games between UvA Base with four midfielders who learned pass selection 
in N games and UvA Base with midfielders who did not learn but only randomly made passes to 
teammates 

N Games Wins-draws-losses Goals 

30 14-5-11 31-24 

5 30 22-2-6 59-23 

10 30 23-2-5 51-17 

30 30 23-1-6 46-15 

50 30 26-0-4 54-19 



5.3 Evaluation Experiments 

In order to evaluate the effectiveness of the learning method described in Sects. 3 
and 4, we have the team of UvA Base with four midfielders trained in the learning 
experiments play 30 games against UvA Base with midfielders who did not obtain 
any learning. The four midfielders in the former team make a deterministic decision 
because parameter T in their policies is fixed at a very low value in the evaluation 
experiments. The midfielders in the latter team randomly make passes to teammates 
because all a)y are fixed to 0. 

Table 1 shows that learning pass selection in only five games (A^ — 5) increases 
the team's goals and decreases the opponent's goals. Therefore, passes by the four 
midfielders contribute to both offense and defense. Furthermore, it is occasionally 
observed in full games that a midfielder selects himself as a receiver and begins drib- 
bling when there is no appropriate receiver to whom the ball can be safely passed. 
Consequently, they learned how to make a decision on whether they should select 
passing or dribbling the ball. 



6 Conclusion 

This research develops a learning method for the pass selection problem of mid- 
fielders in RoboCup Soccer Simulation games. A policy gradient method is applied 
as a learning method to solve this problem because it can easily express the various 
heuristics of pass selection in a policy function. We implement the learning function 
in the midfielders' programs of a well-known team, UvA Trilearn Base 2003. Ex- 
perimental results show that our method effectively achieves clever pass selection 
by midfielders in full games. Moreover, in this method's framework, dribbling is 
learned as a pass technique, in essence to and from the passer itself. It is also shown 
that the improvement in pass selection by our learning helps to make a team much 
stronger. 

In the future, we will apply our learning method to other tasks in a soccer game, 
such as a receiver's selection of the destination point for receiving a pass. A receiver 
must move to receive a pass safely, to receive a through pass from a passer, and 



Policy Gradient Approacli for Learning of Soccer Player Agents 145 

to deceive opponent players. Consequently, the receiver must learn some policies 
and change them depending on the situation. Moreover, an agent becomes a passer 
and a receiver depending on the time and situation. For example, a passer and a 
receiver must change their roles at the next moment to succeed with a wall-pass. 
Accordingly, an agent must change its role and policy at every moment. The next 
step of our work will focus on how agents learn the most desirable selection of roles 
and policies in their current situation. 



Appendix 1: Heuristics from Ui to Us 

We define five heuristics from Ui to Us for evaluating the current position of a 
teammate as a desirable pass receiver. All Uj are normalized between and 10. 

(i) Ui(ax; sx) considers the existence of opponents on the pass course and is 
defined by 

10.0 (if diff > 30 or OppDist > AgentDist or when dribbling), 

9.0 (if 20 < dijf < 30 & OppDist < AgentDist), 

Ui = \ 7.0 (if 10 < diff < 20 & OppDist < AgentDist), (7) 

4.0 (if diff < 10 & OppDist < AgentDist), 

2.0 (else), 

where diff is the angle between the pass direction and the direction to an opponent. 
AgentDist is the distance between a passer and a receiver. OppDist is the distance 
between a passer and the nearest opponent player to the passer (Fig. 4a). If a passer 
cannot get enough information to calculate all three values, we set Ui — 2.0. If a 
passer perceives more than one opponent, values of Ui are calculated for all oppo- 
nents and the smallest of these values is used. 

(ii) U2(ax', sx) evaluates distance min_dist between a receiver and the opponent 
nearest to the receiver (Fig. 4b). It is defined by 

10 (ifm/n_t/wf>30), 

C/2 = 10 - (30.0 - min_dist) /5.0 (if min_dist < 30), (8) 

. 2.0 (else). 

(iii) U^iax; sx) takes into account the number of opponent players around a re- 
ceiver. OpponentsNum is the number of opponent players standing within a IO- 
meter radius of the receiver: 

10 — OpponentsNum (if OpponentsNum < 10), 



f/3 = 



(9) 
0.0 (else). 



(iv) Un{ax\ sx) expresses a heuristics that the nearer receiver to the opponent's 
goal has a greater chance of scoring a goal. It is defined by 



U^: 



1 - goal_dist/ 10.0 (if goal_dist < 40.0) , 
2.0 (else). 
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Fig. 4 Heuristics used for a passer to select a receiver 

where goal_dist is the distance between a receiver and the goalmouth of the oppo- 
nent team. 

(v) U5 (ax; sx) evaluates the degree of certainty of a receiver's position perceived 
by the passer and is defined by 

9.0 (if a receiver and a passer is the same player), 

10.0 X confidence (else), 

where confidence is the reliability of the receiver's position. The value confidence is 
defined by 



Uy 



(11) 



/ CurrentCycle — LastSeeCycle 
confidence = max ( 1 



100 



(12) 



where CurrentCycle is the current time and LastSeeCycle is the latest time when the 
passer saw a receiver. The later the information is, the higher its reliability is. 



Appendix 2: Non-Markov Decision Processes 



In Ref. [14], we defined non-MDPs and proved several theorems even in non- 
MDPs. We show a brief summary of Ref. [14] in this appendix. Non-Markov De- 
cision ProcQSses(Non-MDPs or N-MDPs) are defined by stochastic decision pro- 
cesses without one or some properties in transition probabilities P" , — P(st+i — 
s'\st — s,at — a), expectation of rewards R" , = E[r,^i \s, — s,at —a, St+i — s'], 
and agent policy jt(s, a) — P(a, — a\st — s), where St is the state of the environ- 
ments and at is an action of an agent at time t. An agent receives reward r, at time 
t. Note that whether a learning problem becomes a problem in MDP or non-MDP 
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depends on a system designer's definition of 5, a, and jt{s, a). Non-MDPs are pro- 
cesses where at least one of three quantities depends not only on the current state 
of the environment but also past states or actions. We proved four theorems and two 
corollaries in Ref. [14]. However, we show only Theorem 1 in this appendix. 

MDP is a special case of non-MDP. Theorem 1 shows how the gradient of the 
expected discounted return in non-MDPs is expressed in MDPs. 



Theorem 1 In MDPs, 



dp \— v ^ \-^d7t(s,a) 



9a)/ 



where 



p(jt) 



J2y'''> '■'O'^ 



L r=0 



Q''(s,a) = E 



^ y''rr+k \s,^s, a,^a,n 



k=0 



and 



c/'^(^) = ^y'P(i,=^Uo,7r). 



(13) 

(14) 
(15) 

(16) 



f=0 



p(7r) is the expected discounted return from initial state sq following policy jr. 
2" (s, a) is an action-value function, and d^ (s) is a discounted weighting of states 
encountered starting at 50 and then following policy jr. The expression in (13) agrees 
with the result derived independently by Sutton et al. [18] and by Konda and Tsit- 
siklis [19] for policy gradient learning in MDPs. 
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Genetic Algorithm for Forming Buyer Coalition 
with Bundles of Items in E-Marketplaces 



Anon Sukstrienwong 



Abstract The benefits of buyer coalitions are well-known for electronic market- 
places. However, a few existing buyer coalition schemes over the Internet have fo- 
cused on forming a buyer coalition with bundles of items. This paper presents an al- 
gorithm to form a buyer coalition with bundles of items by using genetic algorithms 
(GAs). The algorithm called GAGroupBuying finds the best disjoint subsets of all 
buyers based on the total utility which addresses the situation where a whole group 
of buyers can be partitioned into smaller sub-groups to obtain more utility than 
they could accomplish in the whole group. The proposed algorithm is compared 
with a previous algorithm called GroupPackageString as shown by Boongasame 
and Sukstrienwong (Emerging Intelligent Computing Technology and Applications, 
pp. 674-685, 2009). The results of GAGroupBuying simulation are found to be sat- 
isfactory with the total discount of a buyer coalition. 

Keywords Genetic algorithm ■ Buyer coalition ■ Bundles of items ■ Coalition 
structure 



1 Introduction 

To date, sellers prefer to put their products on the electronic marketplaces because 
it is a big channel to sell their products in a large number. And, several com- 
mercial websites such as http://buy.yahoo.com.tw, http://www.amazon.com, and 
http://www.staples.com usually offer a volume discount for customers if a num- 
ber of selling is big. In common, buyers prefer to obtain a deduction from the price 
list offered by sellers in return for payment. One common shopping tactic which 
most buyers are likely to make is a group buying because a large group of buyers 
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gains more negotiating power. So, they can advantageously bargain with sellers to 
get a great discount on their purchases. Most buyer coalition schemes form a buyer 
coalition in e-marketplaces to gain the discount for buying a large number of goods 
as shown by Chang et al. [2], He and loerger [4], and Chen et al. [5]. Then, the Com- 
binatorial Coalition Formation scheme of Li and Sycara [3] considers an e-market 
where each buyer places a bid on a combination of items with reservation prices, and 
sellers offer price discounts for each item based on volume. Also, the work of Ito 
et al. [6] presented an agent-mediated electronic market by group buying scheme. 
Buyers or sellers can sequentially enter a market to make their decisions. Tsvetovat 
et al. [7] have investigated the use of incentives to create buying group. Yamamoto 
and Sycara, [8] presented the GroupBuy Auction scheme for forming buyer coali- 
tion based on item categories. Then, the paper of Hyodo et al. [9] presented an 
optimal coalition formation among buyer agents based on a genetic algorithm (GA) 
with the purpose of distributing buyers among group buying site optimally to get 
good utilities. These strategies address the situation where different buyers partici- 
pate in one group to purchase goods at low cost. So, a whole group of buyers can 
advantageously deal with sellers to gain more discount for a large volume of items. 
However, most researchers do not consider forming a group buying with bundles 
of items which addresses the situation where a whole group can be partitioned into 
smaller sub-groups to obtain more utility than they could accomplish in the whole 
group. 

In the corresponding conference paper Sukstrienwong [13] to this paper, fur- 
ther results are found. The proposed algorithm called GAGroupBuying is applied to 
the set of buyers who want to purchase some particular items within several pack- 
ages. The concentration of this paper is to find the best formation of a group buying 
which can give the better group's utility. GAs are a heuristic search scheme based 
on a model of Darwinian evolution. So, there is no guarantee for convergence of 
GAs, but the experiments have shown that the approach is acceptable. The paper is 
divided into five sections, including this introduction section. Section 2 outlines the 
problem of buyer coalition with bundles of items. The detail of the GAGroupBuy- 
ing is shown in Sect. 3. Experimental results of the GAGroupBuying and discus- 
sions are provided in Sect. 4. Finally, conclusions and further work are provided in 
Sect. 5. 



2 Outlines the Group Buying with Bundles of Items 

In this paper, there are some assumptions applying to the group buying with bundles 
of items. First, the group buying is formed under one goal to maximize a buyer 
utility which can be calculated by the discount received by being a member of a 
coalition. Also, the definition of bundles of items is a slightly difference from the 
work of Giirle et al. [10]. In this paper, a bundle of items refers to several items 
together in a package of one or more goods at one price. Additionally, buyers have 
several choices of items, and they have seen the price list of all packages provided 
by sellers before they can place their orders. And, if the package is pure bundling. 
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Sellers 



Package 
numbers 



Product types 

Toilet paper Paper tower Lotion 



Detergent 



Price ( 
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pack of 1 
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pack of 4 
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4 
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- 


2 
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_ 
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12.00 
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36.08 


of 4 
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40.21 




pack of 1 


- 


7.89 




pack of 3 


pack of 2 


18.93 




- 


pack of 9 


47.21 




- 


pack of 1 


8.00 


of 4 


pack of 3 


- 


40.64 


of 4 


- 


- 


59.76 


of 2 


pack of 3 


pack of 3 


52.00 



Table 2 An example of buyer's orders with the reservation prices 



Buyers 



Buyer's order (number of items x (reservation prices $)) 



Toilet paper 



Paper tower 



Lotion 



Detergent 



bo 
bi 
hi 
b3 



2x(9.1) 



1 X (9.1) 



4 X (10.0) 



1 X (6.55) 



3 X (6.0) 



2 X (5.95) 
1 X (6.0) 



the average price of each item will be cheaper than the price of the same product 
in a single item package. An example that is given to describe a motivation for this 
paper is described below. 

There exist several commercial websites providing some attractive products 
with the special prices. Suppose there are four types of products selling in an e- 
marketplace which are toilet paper, paper tower, lotion, and detergent. Sellers make 
a price list for their products; both single item and bundles of items, as shown in 
Table 1. For instance, package number 3 of seller 1, denoted Pj, comprises of two 
packs of toilet paper and four packs of paper tower. This package is sold at the spe- 
cial price of $59.76. However, a single pack of toilet paper and a four pack of paper 
tower are sold separately at the price of $12.00 and $40.21 by seller 0, as seen in 
package number 1 and 3 of seller 0, respectively. A symbol ' — ' appearing in Table 1 
means that sellers do not put a specific product in that package number. 

Suppose there are some buyers requesting to purchase some products listed in 
Table 1 . Due to buyer demands and economic problems, some buyers do not want 
to purchase a whole package by their own, so they come to participate in the group 
buying with the aim of obtaining their requested products at a good price on the pur- 
chasing. Suppose there are four buyers joining into a particular group. Each buyer, 
denoted as b,„ where < m < 4, places some orders to specific items with reser- 
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vation prices as shown in Table 2. For instance, a buyer bo wants to get 2 packs 
of toilet paper at a reservation price of $9.10 of each and a pack of lotion at a 
reservation price of $6.55. The reservation of each means the price for particular 
item that buyers are willing to pay. Also, a symbol '—'in Table 2 under each prod- 
uct means that a buyer has no request to buy. If a buyer called bo goes directly 
to purchase those products by his own, the total cost that bo needs to pay for two 
packages of Pq and one package of P3 from seller so is 12.0 *2 + 7.89 = 31.83 
dollars, while bo is willing to pay totally for those items at 2 * 9.1 + 6.55 — 24.75 
dollars. It seems to be impossible for bo to purchase all items along. So, bo would 
come to participate in a group with the aim of obtaining better prices on the pur- 
chasing. 



3 Genetic Algorithms for Buyer Coalition with Bundles of Items 
3.1 The Basic Concept ofGA 

Genetic algorithms (GAs) are a heuristic search algorithm based on the evolutionary 
ideas of natural selection. The basic mechanism of GAs is designed to simulate pro- 
cesses in nature necessary for evolution. GAs often apply to find the optimal solution 
to the problem by manipulating a population of solutions. The manners of problems 
need to be encoded in chromosomes for distinguishing good solutions form bad 
ones. Only good individuals of the population survive to the next generation while 
bad individuals are eliminated from the selection process. The genetic algorithm be- 
gins with generation with the completely random population size M. There exist 
two operations to perform during one generation, crossover and mutation. During 
the run, the given individual might be mutated and crossed within single generation. 
Note that the GAs work on fixed-length character strings. The simplest algorithm 
represents each chromosome as a bit string which are strings of I's and O's. In gen- 
eral, GAs search the space of possible character strings in an attempt to find good 
string based on fitness value of strings. 

There are few steps in preparing to use the genetic algorithm of fixed-length 
character strings to solve a problem as shown by Koza [11], genetic programming 
on the programming of computers by means of natural selection. 

1 . Selecting a representation that facilitates the solution of the problem by means 
of the genetic algorithm. 

2. The fitness value of each element, which could be the objective of solution, is 
used to distinct good and bad individuals from the population. In general, the 
fitness measure is inherent in the character string it encounters. 

3. For setting up the genetic algorithm, primary parameters for controlling the ge- 
netic algorithm are the population size (M) and the maximum number of genera- 
tions to be run (Gen). Secondary parameters, the crossover probability (pc), and 
the mutation probability (pm) are required to create new population. 
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4. Randomly create an initial population of individual fixed-length character 
strings. Evaluate the fitness value of each individual in the population. Create 
a new population for the next generation based on fitness value by randomly 
applying two operators, crossover operator and mutation operator which are de- 
scribed below. 

• Crossover operation: Create two new offspring from two existing mem- 
bers in the population by recombine randomly chosen substring from selected 
members. 

• Mutation operation: Create a new offspring from an existing member in the 
population by randomly mutating the character at one position in the fixed- 
Length string. 



3.2 Problem Formalization 

Let B — {bo, bi, b2, . . . , bn) denote the collection of buyers. Each buyer wants to 
buy several items in the e-marketplaces (Figs. 1, 4, 7). There is a set of seller 
S = {so, si, S2, . . . , Si}. One seller called si has made special offers within a set of 
packages, denoted as Pi — {Pp P' , P?, . . . , PJ'}, where k is the maximum number of 
packages. A price per item is a monotonically decreasing function when the size of 
the package is increasing big, an each package is associated with the set of prices, 
denoted Pricci — {pricey price^ price?, . . . , pricej'}. Each package PJ' is comprised 
of multiple items associated by a vector {g''''o, g''''i, g''''2, ■ ■ ■ , g''''j)- If a goods 
named g'-'^j is not available in the package, then g''''i — 0. Each bn needs to buy 
some items offered by sellers, denoted as Qn = {q"o, q"i, q"2, ■ ■ • , q"j }, where q"j 
refers to the quantity of items gj of buyer n. If q"j = 0, it means that bn does have 
no request to purchase gj. Also, bn places the reservation price for each particular 
goods associated with Qn, denoted as Rsn — {rs"o, rs"i, rs"2, • ■ ■ , rs"j } where rs"h 
> 0, 1 < h < j. As stated earlier, there is a situation where a whole group of buyers 
can be partitioned into smaller groups to receive a better utility than they could get 
by joining everyone in the whole group. A group of buyers can be divided into small 
groups of buyers called sub-coalitions, denoted as Cn which Cn C B and Co n Ci 
n ■ • ■ n Cn = 0. There is a situation where a coalition structure CS is a partition 
of B of which each buyer of B belongs to exactly one coalition and some buyers 
may be alone in their coalitions. For example, if B = (bo, bi, b2, bs }, there are 15 
possible coalitions: (bo), {bi}, {b2}, {hj}, {bo, bi}, {bo, b2}, {bo, bs), {bi,b2}, 
{bi,b3}, {b2,b3}, {bo, bi,b2}, {bo, bi,b3}, {bo, b2,b3}, {bi, b2, b3}, and {bo, bi, 
b2, b3 ). From the work of Sandholm et al. [12], there are 2" — 1 possible coalition 
structures for n buyers. This is simply the number of subsets of B (not counting the 
empty set). So, for buyers = 4 there are 2^ — 1 = 16— 1 = 15 possible coalition 
structures: {{bo}, {bi}, [hj], {h^}}, {{bo}, {bj}, {b2,b3)), {{b2}, {b3), {bo,bi}, 
{{bo}, {b2}, {bi,b3}}, {{bi}, {b3}, {bo,b2}}, {{bo}, {b3}, {bi,b2}, {{bi}, {b2}, 

{bo,b3}}, {{bo}, {bl,b2,b3}}, {{bo,bi}, {b2,b3}}, {{bi}, {bo,b2,b3}}, {{bo, 
b2}, {bl,b3}}, {{b2}, {bo,bi,b3}}, {{bo,b3}, {bi,b2}}, {{b3}, {bo, bi, bz } }, and 

{bo, bi,b2,b3}. 
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Fig. 1 A coalition structure graph of 4 buyers 

Fig. 2 An example of {bo.bi} <P2iP2iP*o> {bj.bj} <P3iP2oP*i> 

BuyerChromosome of { {bo, 
b|}, |b2,b3)} 

3.3 Problem Representation for Forming Buyer Coalition 

In this paper, the BuyerChromosomes are designed to facilitate genetic algorithm 
processes. A BuyerChromosome is the chromosome of sub coalition together with 
a sequence of random packages. For instance, suppose a set of buyer B = {bo,bi,b2, 
b3 } is randomly divided into two sub-groups, Co = {bo, bi } and Ci — [bj, bs } . And, 
the chosen packages for sub-coalitions are arranged in the form of <P'j>^, where L 
is the maximum number of packages. An index i refers to the number of sellers, 
and an index j means the number of the packages provided by seller Si. Suppose the 
algorithm has randomly chosen <P iP iP o> for Co and <P iP oP i> for Ci. 
Then, a BuyerChromosome of { {bo, bi ), {b2, bs } } is built as shown in Fig. 2. 

From Fig. 2, Co buys two packages of P2 from seller so and one package of 
P4 from seller so. And, Ci buys one package of P3 and P4 from seller si and one 
package of P2 from seller so. In common, if there exist n buyers who want to buy 
products from i sellers offered to sell k different packages, then the possible number 
of BuyerChromosomes is (2"' —m — l)(i)(k'^), where L is the maximum number of 
packages of each BuyerChromosome. 



3.3.1 Fitness Measurement 

In this paper, the fitness value to each BuyerChromosome in the population is the 
total utility of a coalition. The utility of a coalition is a discount obtained from 
sellers by purchasing goods. The sum of all buyers' reservations of the sub-coalition 
denoted as reservation(Cn) is shown as follows: 

k 



reservation{Cn) — / / > 



rs j * q 



(1) 
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Parentl Parent! 

{bo.bl) <P2lP2iP^o> {b2,b3} <P^P2oP*l> (bo,b2,b3) <PioP2iP3o> {bi} <P2oP2i> 

^ i 



{bo,^} <P2iP2iP4oi{b2,b3} <P^P2oP*i> I |{bobi}<I"oP2iP5o>{bi}<P2oP2i> 



{bo,b2jbs}<P2iP2iP«o> 
Offspringl 



r 



{bo,bi}<P'oP2iP3o> 
Offsprings 



{bo) <P2lP2iP«o> {bj.bj) <P3iP2oP*l> {bo} <PloP2lP3o> {bl} <P2oP2i> 

Offspring! Offspring4 

Fig. 3 An example of buyer crossover of Co at buyer-crossover point = 2 



Parentl Parentl 

(bo.bi) <P2iP2iP'o> {b2,b3} <P3iP2oP'i> {bo,b2,b3) <PioP='iP'o>{bi} <P2oPl> 

7^ V 

{bo.bi) <P2iP2iP*o> {b2,b3} <P3iP!ii {bo,b2,b3} <PioP2iP^o> (bl) <P2oEQ21e 
Offspringl Offspringl 

Fig. 4 An example of package crossover of C| at package-crossover point = 2 

And the total amount paid by a sub-coalition, called Cn, to sellers denoted as 
AmountPaid(Ca) is 



AmoutPaidiCa) — / Price': . 

7=0 



(2) 



The total utility of Cn, denoted as Utility(Cn), is the defined as follows: 

Utility (Cn) — reservation{Cn) — AmountPaid(Cn)- (3) 

So, the fitness function of each individual in the population of the coalition is 
mathematically defined as follows: 



Fitness _f unction — >^ Utility{Cn) ■ 



(4) 



3.3.2 Breeding a New Population of BuyerCliromosomes 



For each new solution to be produced, a BuyerChromosome is selected for breed- 
ing based on its fitness value. By producing new offspring, in the GAGroupBuying 
methods of crossover and mutation are described below. 
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1. Crossover operation: This operator creates new offspring from existing par- 
ents of current generation in the population by having crossover operation at a 
random crossover point. Suppose two selected BuyerChromosomes in crossover 
with L = 3 are {bo, bi }<p2iP2ip4o>{b2, b3}<P^PV^i> and {bo, b2, 
b3}<P'oP^iP^o> {bi )<P^oP^i>- The parents will be split at the crossover 
point into two substrings, a crossover fragment and a remainder. In this paper, 
there are three types of crossover operation which are buyer crossover, package 
crossover, and sub-coalition crossover. 

• Buyer crossover: Suppose the GAGroupBuying randomly selects Co with a 
buyer-crossover point — 2 for both parentl and parent2 shown in Fig. 3. Two 
possible resulting of recombining the crossover fragment and the remain- 
der of two substrings are {bo, hj, b3}<p2iP2,p4o>{b2, b3}<P\p2oP'^i> 
and {bo, bi }<PVP^iP^o>{bi } <P^oP^i>- As the result shown, these new 
BuyerChromosomes are not completely done because each new BuyerChro- 
mosome has the problem where Co n Ci ^ 0. So, these new BuyerChro- 
mosomes are required to split into new BuyerChromosomes. Finally, four 
new BuyerChromosomes are generated which are {bo, b2, b3}<P^iP^iP''^o>. 
{bo}<p2iP2iP4o>{b2, b3}<P^p2oP^>, {bo, bi } <pioP2iP^o>, and 
{bo}<pioP^P^o>{bi)<PV^>. 

• Package crossover: If the GAGroupBuying chooses to operate with the pack- 
age crossover, the sub-coalition of current BuyerChromosome and a package- 
crossover point is randomly selected. Suppose the GAGroupBuying randomly 
select Ci with the buyer-crossover point — 2 for both parentl and parent2. 
Two possible resulting are {bo, bi}<P^iP^iP'*o>{b2, b3}<P^iP^i> and {bo, 

b2,b3}<ploP'lP'o>{bl}<PWl>- 

• Sub-coalition crossover: This operator chooses the sub-coalition of each par- 
ent and replaces it into the other parent. Suppose Co of parentl and Ci of 
parent2 are randomly selected. Then Co of parentl is replaced into the po- 
sition of Ci of parent2. Also, Ci of parent2 is replaced into the position of 
Co of parentl. This process generates two new strings, {bi )<P^oP^i>{b2, 
b3}<P^p2oP^> and {bo, b2, b3)<pioP^P^o> {bo, bi}<p2ip2ip4o>. 
However, the second string requires to be regulated because bo is allocated in 
two sub-coalitions. So, {bo, b2, b3)<P'oP^iP^o> (bo, bi }<p2iP2ip4o> is 
split into new strings which are {bo, b2, b3}<P'oP^iP^o>{bi }<p2iP2iP^o> 
and {b2, b3}<P'oP^iP^o>{bo, bi}<p2iP2ip4o>. The result of the sub- 
coalition crossover is shown in Fig. 5. 

2. Mutation operation: This operator creates a new offspring from best existing 
BuyerChromosomes in the population. The mutation process begins by randomly 
selecting a sub-coalition and choosing a number as a mutation point. The chro- 
mosome at this mutation point is randomly changed. Suppose the random select- 
ing BuyerChromosome is {bo, bi }<p2ip2ip4o>{b2, b3}<P^iP2Qp4i>. And, a 
sub-coalition Co with mutation point — 1 are chosen in random. Then, the pos- 
sible resulting of mutation operation is shown in Fig. 6. 
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Farentl Parent! 

fbn.biKP^P^,P*n>fb,.b,) <P3iP2oP«i> {bo.b2,b3} <P'oP^iP^o>fbi)<PnP^,> 

^i 



fbi)<Kp='i>i'b2,b,><P^,P'nPS> I |fbn,b9.b,l<P'nP^iP^n>fbn.bn<P^iP^iP*n> 



r 



fbil<P^oP='l> fb2.b;i) <P3,P2oP*i> {bo,b2,b3} <P'oP='iP^o >fbil<P^iP='iP«o> 
Offspringl Offspring! 



Fig. 5 An example of sub-coalition crossover 



{b2,b3} <P'oP^iP^n> fbn.b 1 ) <P^iP^,P^n> 
Offsprings 



Fig. 6 An example of 
mutation operator 



Parent {bo.bi}<P2iP2iP*o>{b2.b3}<P3iP2oP*i> 



r 



Offspring {bo.bi} <P2iP^P*o> {b2,b3) <P3iP2oP*i> 



The GAGroupBuying keeps generating a new population for each generation 
until it gets the maximum number of generations to be run. At last, the best Buyer- 
Chromosome, which is possible to yield the highest profit to the group buying based 
on the fitness value might be found. Suppose the best BuyerChromosome of B = 
{bo, bi, b2, b3} is {bo}<P'oP^P^o>{b2, bs} <P^p2oP'^l>. Each sub-coalition 
has its utility which is calculated by Eq. 4. This buyer coalition is formed without 
buyer bi , so the total utility of the coalition is calculated by Eq. 5. 



TotalJJtility — ^ Utility(Cn) 



(5) 



4 Experimental Results and Discussions 



The experimental results of the proposed algorithm are derived from a simulation 
which has implemented more than 4000 lines of C-n- program. It is run on a Pen- 
tium(R) D CPU 2.80 GHz, 2 GB of RAM, IBM PC. The primary parameters for 
controlling the proposed algorithm, as seen in Table 3, are the population size (M), 
the maximum number of generations to be run (Gen), the crossover probability (pc), 
and the mutation probability (pm)- The other parameters are for setting sellers and 
buyers used in the simulation. There are two sellers (NumOfSeller = 2) which are 
arbitrarily set to sell several products in packages, while there are ten buyers (Nu- 
mOfBuyer =10) requesting to buy several items with the reservation prices. The 
maximum number of items a package is 5 (MaxItemPerPackage — 5). Some pack- 
ages listed by sellers are pure bundling packages, but some packages are single-item 
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Table 3 The summarized parameters 


1 for the GAGroupBuying simulation 






Constants 




Detail 








Range 


M 




Population size 








1000 


Pc 




Crossover probability 








0.95 


Pm 




Mutation probability 








0.05 


NumOfSeller 




No. of sellers 








2 


NumOfBuyer 




No. of buyers 








10 


MaxltemPerPackage 


Max number of items 


per 


packagi 




5 


MaxNumOfPackagePerSeller 


Max number of packa 


iges 


for each seller 


10 


Table 4 A price 


list of two sellers 












Sellers 


Package 
numbers 


Product types 








Prices ($) 




A B 


C 




D 









1 


_ 




_ 


1000 


So 


1 


2 1 


- 




- 


2690 




2 


4 


- 




- 


3400 




3 


- 


1 




- 


900 




4 


- 


4 




1 


4200 







1 


1 




- 


1700 




1 


- 


1 




1 


1740 




2 


1 


- 




- 


900 


Sl 


3 


4 


- 




- 


3580 




4 


1 


- 




1 


1970 




5 


- 


- 




1 


955 




6 


- 


- 




5 


4250 




7 


1 


1 




1 


2750 




8 


1 1 


- 




1 


2700 



packages. All price lists offered by two sellers is shown in Table 4. And, there are ten 
buyers who want to participate in the group buying. An example of buyers' orders 
with reservation prices are listed in Table 5. 

The experimental result received from the simulation is shown in Table 6. It 
shows that the GAGroupBuying finds the better total utility than the GroupPack- 
ageString. The group buying can be formed by separating buyers into two sub- 
coalitions, {bo, bi, b2, b3, b4, be, bg} and {bs, b?, bg}. The best of BuyerChro- 
mosomes found by the GAGroupBuying during the simulation is {bo, bi, b2, bs, 
b4, be, b8}<p2o>{b5, b?, b9}<PSP'*o> with the total utility of $100. The num- 
ber of buyers who can purchase the items is 9. And, the average total utility earned 
from GAGroupBuying is 89.12. A buyer who fails to join with any sub-coalitions 
is bo, so it is removed out of the group buying. The result is also shown in graphs 
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Buyers 


Buyer's order (number of items x reservation price $) 








A B 


C 




D 


bo 


1 X (860) 


- 




- 


bi 


- 


- 




1 X (860) 


b2 


- 


2 X (855) 




- 


b3 


- 


1 X (860) 




- 


b4 


- 


- 




1 X (855) 


b5 


1 X (860) 


- 




- 


bfi 


- 


1 X (865) 




- 


bv 


1 X (860) 


- 




- 


bs 


- 


1 X (860) 




- 


b9 


2 X (855) 


- 




- 


Table 6 Experimental results 


GAGroupBuying 


GroupPackageString 








Number of 


Total buyers Avg. of total 


Total buyers 


Avg. 


of total 


sub-coalitions of 


who can utility (10 runs) 


who can 


utility (10 runs) 


(the best result) 


purchase the 
items 


purchase the 
items 






||bo, bi,b2,b3, 


9 89.12 


10 


48.14 




b4. bg.bg}, {bs, 










bv.b,)) 











the comparison result of GroupPackageString and GroupPackageString. Brief jus- 
tifications for the values of this graph are M — 1000, pc — 0.95, and pm = 0.05. 
At the beginning of the graph, when the number of generations is low, both algo- 
rithms perform almost similar. However, when the number of generations is about 
550, the curve of the GAGroupBuying graph increases rapidly over the cure of the 
GroupPackageString graph. It is because the quality of the GAGroupBuying for 
forming a buyer coalition improves. One prominent characteristic designed for the 
GAGroupBuying is that buyers with bad reservation be able to identify, and they 
possible to be removed out of the group buying. So, the average total utility earned 
by the GAGroupBuying is higher than the average total utility earned by Group- 
PackageString. 



5 Conclusions and Future Work 



The paper presents an algorithm called GAGroupBuying for forming a buyer coali- 
tion with bundle of items. The aim of this algorithm is to maximize the total utility 
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Fig. 7 The comparison 


result 
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of the group buying. The algorithm based on GA works by randomly partitioning 
a whole group of buyers into smaller sub-groups with the aim of obtaining more 
utility (or discount) than they could receive in the whole group. From the experi- 
mental result, the GAGroupBuying is able to find better utility than the GroupPack- 
ageString, but the GAGroupBuying requires a bigger number of generations to be 
run. And, there are some restrictive assumptions for this algorithm. The buyer coali- 
tion is formed concerning only the price attribute, and buyers of the coalition cannot 
buy the bundles of items by themselves. All buyers do not know each others in the 
coalition. Sellers in electronic marketplaces can supply unlimited items of any prod- 
ucts. If the package is pure bundling, the average price of each item will be cheaper 
than the price of a single item package. These restrictions can be extended in future 
researches. 
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Inside Virtual CIM 

Multi-agent Based Resource Integration for Small 
to Medium Sized Manufacturing Enterprises 



Ning Zhou, Sev Naglingam, Ke Xing, 
and Grier Lin 



Abstract Worldwide cooperation among manufacturing companies is increasingly 
gaining importance to face emerging challenges in manufacturing. The traditional 
Computer Integrated Manufacturing (CIM) systems cannot satisfy the needs of 
global market as they are deployed only within an enterprise. Therefore, a more 
flexible and comprehensive integrating methodology is required to overcome dis- 
tance barriers, facility sharing problems and communication obstacles. These issues 
lead to the concept of Virtual CIM (VCIM). In this paper, the limitations of cur- 
rent agent based implementation of VCIM concept are analyzed. It also describes 
approaches to address those limitations and propose further development on Agent 
based Resource Scheduling Process in Small and Medium Enterprises in VCIM 
network. 

Keywords Virtual CIM ■ VCIM ■ Small and medium manufacturing enterprises ■ 
Multi-criteria selection ■ Agent based systems 



1 Introduction 

Manufacturing enterprises face challenges from worldwide competitors and strug- 
gle to extend their business globally in today's economic climate. Therefore, these 
enterprises tend to use fully integrated manufacturing systems so that they can have 
the capabilities to rapidly respond to constantly changing customer requirements 
and produce high quality products in shortest possible time with lowest possible 
cost. However, Small and Medium Manufacturing Enterprises (SMMEs) are hav- 
ing difficulties in achieving this flexibility and competing with large companies, 
since SMMEs often do not have enough resources. By integrating manufacturing 
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resources of many partner enterprises (which may be located at different regions), 
these enterprises can form a globally integrated SMME network and achieve a com- 
petitive edge [1]. 

Virtual Computer Integrated Manufacturing (VCIM) is a new concept for Com- 
puter Integrated Manufacturing (CIM). In 1997, it was proposed by Lin in his 
keynote in Singapore. He stated the VCIM concept would be the future evolvement 
of CIM [1]. The main improvement for the word "Virtual" is to address the lim- 
itation of CIM definition. In the traditional concept, CIM is often limited within 
an enterprise. However due to the global competition and collaboration, a solu- 
tion is needed to expand CIM to a much wider border. Thus Virtual CIM (VCIM) 
was defined as a network of interconnected global CIM systems, extends traditional 
concept of CIM from a local and centralized company to world-wide cooperation 
[2]. Virtual Computer Integrated Manufacturing (VCIM), which is a network of in- 
terconnected global Computer Integrated Manufacturing (CIM) systems, extends 
traditional concept of CIM from a local and centralized system to worldwide co- 
operation [2]. Today, Companies are expanding their business boundaries locally 
and internationally, merging or co-operating with others across geographical demar- 
cation. VCIM aims to unite and integrate all activities in an enterprise or a network 
of enterprises to share resources and management objectives through information 
integration, in a cohesive manner to work as a seamless global CIM system. In a 
VCIM system, manufacturing resources may belong to different enterprises or be 
located at different areas, but all have intention of working together in an integrated 
manner. After receiving a product order, the VCIM system schedules and organize 
distributed resources as a temporary production system based on working status in- 
formation of the resources in real time. As this temporary system disappears when 
purpose are fulfilled, compared to the tight connection of manufacturing resources 
in traditional CIM systems, this transient system with VCIM is in a virtual status. 
To describe this status viewpoint, the word Virtual is used in the concept of VCIM 
(Virtual CIM). 

The effort for global collaboration in manufacturing industry never ends. VCIM 
aims to integrate all manufacturing resources from different entities in different lo- 
cation, and make them work together as a seamless global CIM system [3]. When 
receiving a customer order, this system searches among available resources and 
schedules a best path for production schedule. Those resources in the schedule will 
form a temporary system to fulfill that customer order. The "virtual" comes from 
the concept of Virtual Enterprise (VE) studies [4-6]. VE is defined as a collection 
of companies that are dynamically composed when needed and dissolved after its 
goal is completed [7]. VE intends to integrate all elements from raw materials in 
a supply chain as a whole to provide to final customers [8]. However, VCIM has 
a much wider scope. It has many correlations with other integration concepts such 
as Intelligent Manufacturing System [9], Holonic Manufacturing System [10]. The 
aim of the VCIM is to establish cohesive connections among those manufacturing 
resources. Those resources are often in the form of Small to Medium Manufacturing 
Enterprises (SMMEs). VCIM is expected to give those SMMEs enhanced competi- 
tiveness in Global competition [11]. 
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This article is the revised/extended work based on the paper presented in 2010 
IMECS conference [12] and is divided by two parts. The first part reviews the back- 
ground of VCIM research project. The second part analyzes the current develop- 
ment with Agent-based approaches to build the proposed VCIM system, including 
the limitations in current system design and potential issues for the system imple- 
mentation. In the second part, possible solutions to above limitations are discussed. 
Major changes to current architecture will be demonstrated. 



2 Current Development and Limitations 

SMMEs play a significant role in pioneering new technologies, markets, and cre- 
ation of knowledge based industries, all of which are important for future growth 
and jobs of many countries. In addition, they are often characterized by niche spe- 
cialist markets in which they have expertise. They succeed by providing high levels 
of responsiveness and personalized service. In many instances, they can offer lower 
priced products compared to other large enterprise, because SMMEs have less over- 
heads and their labour and management force are the same. However, when SMMEs 
grow and develop a typical functional structure, they need to develop efficiencies 
within their total processes to remain competitive. Many growing SMMEs see their 
profit margin drop as these inefficiencies mount. 

Global competition and today's open markets are driving the enterprises to in- 
troduce high quality products and services economically and efficiently. A VCIM 
system is a strategic move that requires manufacturing enterprises to establish close 
relationships in order to exploit each other's core competencies for the betterment 
of the SMME network. 

Participation in a VCIM is especially challenging for SMMEs. Since, the VCIM 
activities involve complex operations and these participants of a VCIM system are 
distributed worldwide, the SMMEs must overcome global boundaries in terms of 
distances, time, regulatory constrains, as well as cultural and political differences, 
for reaching mutually-beneficial agreements on how to optimize the customer order 
fulfillment process. In addition, some of these partnerships are dynamic and becom- 
ing virtual representing the transient status of the collaboration [3]. 



2.1 Agent Based System Architecture 

To optimize resource sharing and to provide a dynamic integration, an agent-based 
VCIM architecture has been developed. In the agent based VCIM system, three cat- 
egories of agents have been identified [4]. These agents include Facilitator Agents, 
Customer Agents, and Resource Agents. Facilitator Agents are designed to act as 
coordinators to route the information flow across the VCIM agent community. Cus- 
tomer Agents are designed to provide interfaces for customer to participate in the 
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VCIM system. Finally, Resource Agents are designed as agent interface to encapsu- 
late distributed manufacturing functional entities and connect them with the agent 
community. The functionalities and responsibilities are described in earlier VCIM 
research [1, 3-5]. 

In VCIM agent-based architecture, all the agents are connected to the Internet. 
Facilitator Agents need to register to Customer Agents and Resource Agents need 
to register to Facilitator Agents. All the communication is delivered via the Internet 

[1]. 

After receiving the customer order, the Customer Agent passes the order to a 
Facilitator Agent. The Facilitator Agent then works together with those Resource 
Agents registered to it and make an optimized production schedule [1]. 

The optimized production schedule is defined as the cheapest schedule with 
shortest duration time while it satisfies the customer's required due date/time and 
delivery destination [3, 4]. 

By connecting a Facilitator Agent, a Customer Agent and many Resource Agents 
through the Internet, these agents can form a basic multi-agent VCIM system. Nev- 
ertheless, a real VCIM system includes many Facilitator Agents, Customer Agents 
and Resource Agent while the functionalities of Resource Agents may include de- 
sign, manufacture, delivery, material supply and others [5]. 



2.2 Current Limitation 

Current Agent-based VCIM architecture still need improvement, because we have 
found three major limitations that prevent the Agent-based VCIM architecture into 
real practice. Those are: 

1 . Network limitation 

The VCIM agents reside across the boundaries of many enterprises. According to 
past research [6], the agent communication between distributed locations is often 
unreliable. Thus when performing a resource scheduling across the VCIM network, 
the big number of VCIM agents, extensive volume of exchanging messages, limited 
bandwidth and unreliable nature of the Internet will slow down the whole process. 
Unlike a faster and more reliable Intranet inside a single organization, the VCIM 
system must use the Internet more wisely with less communication volumes and 
more flexible mechanisms. 

2. Agent selection limitation 

The VCIM network is designed to be dynamic. This means any time a new VCIM 
agent may join or quit the network. Current architecture does not address the mech- 
anism how a Customer Agent finds suitable Facilitator Agents and selects the most 
suitable one among them. For a particular customer order, the most suitable Facilita- 
tor Agent means the Facilitator Agent who most satisfies the Customer Agent with 
its offer content and other comparable factors. 
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3. Multiple criteria selection limitation 

When making selection from proposed work schedule or manufacturing resources, 
current architecture only uses two factors: cost and time [5]. While in real situation, 
many other factors need to be considered, such as: quality, friendship, credit, and 
delivery reliability [7-11]. Lack of a systematical multi-criteria selection method 
will limits us reaching the optimal result. 



2.3 Research Focus 

The research focus is to address the above limitations and give possible solutions to 
enhance the functionality of the Multi-agent based VCIM. Therefore the focus can 
be divided into three parts. 

1. For network limitation, messages exchange over the network must be mini- 
mized. More efficient mechanisms need to be used to optimize message flow and 
lower network overhead and latency. This part will mainly focus on a redesign of 
current agent network architecture. 

2. For agent selection limitation, the way for agent communication needs to be 
improved. This part will focus on changes to agent negotiation and agent commu- 
nication protocol. Detailed steps from customer order to final delivery need to be 
specified. 

3. For multiple criteria selection limitation, multi-criteria needs to be imple- 
mented in two selection processes. First process is for a Customer Agent to find 
and select the best suitable Facilitator Agent for a particular order. Second process 
is for a Facilitator Agent to find and select the best suitable Resource Agent. This 
part will focus on solving the limitation in these two selection processes. 



3 Agent-Based Architecture and Resource Scheduling Process 

This research explores on the revolution of VCIM resource scheduling process and 
there are three major changes in system architecture, communication protocol and 
decision making process. 

To improve the performance and functionality, an artificial intelligent decision 
making process is needed to be integrated to the Multi-agent framework. A multi- 
criteria approach is also needed to be integrated into the agent searching and selec- 
tion process. According to the research focus addressed before, the new improve- 
ments over the new VCIM Agent Architecture and Framework based on three parts: 

1 . Registry Service 

2. VCIM Agent Communication Process 

3. Multi-criteria selection Integration 
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Fig. 1 New VCIM architecture 

Figure 1 shows the new designed VCIM architecture. All small to medium sized 
manufacturing enterprises are connected to the Internet, forming a virtual collab- 
oration network. The VCIM Agents reside in each SMME. Customers use their 
computer to submit orders to the Customer Agent through the Internet. The new 
Public Directory also connects directly to the Internet. It stands alone, not within 
any SMME. And it holds all basic information of each VCIM agent and allows 
queries for agents. The agents inside a SMME will communicate in ACL (Agent 
Communication Language). The agents in different SMMEs will communicate in 
HTTP/SOAP in order to pass enterprise firewalls. 



3.1 Registry Service 



In order to improve the agent communication over Internet, a central registry like 
database that holds all the information of each agent across the whole interconnected 
VCIM network needs to be created to ease the multi-criteria agent searching. Here 
we define it as a Registry Service. A special agent called Directory Agent is created 
to provide this service. 

Figure 2 shows the improvement to current connection structure. We can see that 
along with other agents, the new Directory agent is connected to the Internet. They 
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Fig. 2 Connection structure of a base VCIM system (adapted and modified [5]) 

virtually link together to form a Virtual CIM (VCIM) network. The VCIM network 
is dynamic because any time a VCIM agent may join or quit. When a new agent 
joins in the VCIM network, it needs to register itself to the Directory. When a new 
agent quits in the VCIM network, it needs to de-register itself from the Directory. 
When an agent wants to search for another agent, it submits criteria-based queries to 
the Directory and gets result from the Directory. In old implementation, when a new 
agent joins in the VCIM network, it needs to register itself to all Facilitator Agents. 
When a facilitator wants to find suitable Resource Agents, it needs to make queries 
to all Resource Agents. The advantage of a directory service is obvious. Not only 
can the communication volume be minimized, but also better search response time. 
Figure 3 shows the improvement in current information flow. The messages are 
transported among different types of agents. All agents need to communicate with 
the new Directory Agent for functions like Registering, De-registering, and Search. 
The figure also shows that Customer Agent needs talk to Facilitator Agent and Fa- 
cilitator Agent needs talk to Resource Agents. The communication volume among 
them is greatly reduced, because the Directory can pre-select suitable agents for 
them to talk to. In old architecture, the Customer Agent sends requests to all Fa- 
cilitator Agents. In new architecture, the Facilitator Agent finds suitable Resource 
Agents that can provide the parts and only send requests to them. 



3.2 Agents Based Resource Planning 



When a Facilitator Agent receives a request from a Customer Agent, it generates all 
possible production schedules. Then it finds the best suitable production schedule. 

Table 1 shows the each step when a customer order comes in. When a Facilitator 
Agent receives a request from a Customer Agent, it divides the order into subtasks 
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Fig. 3 Information flow in a basic VCIM system (adapted and modified [5]) 

Table 1 Steps of recourse scheduling for a single order 

Stepl When a Facilitator Agent receives a request from a Customer Agent, it 

divides the order into subtasks. 

Step2 The Facilitator Agent sends a request to Directory Agent to find Resource 

Agents that can perform the subtasks. 

Step3 The Directory Agent receives the request and search in its own database. 

Then return the result to the Facilitator Agent. 

Step4 The Facilitator Agent sends a request to Resource Agents that can perform 

the subtasks. 

StepS The requested Resource Agents generate all possible subtask schedules and 

return the result to the Facilitator Agent. 

Step6 The Facilitator Agent generates all possible production schedules 

according to received subtask schedules from the Resource Agents. 

Step7 The Facilitator Agent finds the best suitable production schedule through a 

multi-criteria method, and returns the result to the Customer Agent. 

StepS After the Customer Agent receives the order quotations, it selects best 

suitable Facilitator Agent through a multi-criteria method and confirms the 
order to that Facilitator. 



and sends a request to Directory Agent to find Resource Agents that can perform the 
subtasks. Then the Facilitator Agent sends requests to Resource Agents returned by 
Directory. The requested Resource Agents generate all possible subtask schedules 
and return the results to the Facilitator Agent. The Facilitator Agent generates all 
possible production schedules according to received subtask schedules from the Re- 
source Agents and finds the best suitable production schedule. After the Customer 
Agent receives the order quotations, it selects best suitable Facilitator Agent based 
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on their quotation content and other considerable factors. After the Facilitator Agent 
receives the order confirmation, it sends subtask confirmation to Resource Agents 
according to the subtask schedule. After the Resource Agent receives the subtask 
confirmation, it starts the production process. 

Multi-agent communication protocol needs to be defined here to support above 
agent selection and resource scheduling. In this system, when a customer submits 
a product order, a VCIM agent search and resource scheduling process is initiated. 
Protocols to support agent communication linkage and information exchange need 
to be defined. For example. Fig. 4 shows an agent communication protocol used 
for Customer Agent to search for suitable Facilitator Agents. Here we use Request 
Interaction Protocol (RIP). The Customer Agent first sends a search request to the 
Facilitator Agent. The Facilitator Agent replies with either refuse or agree. If agree 
the Facilitator Agent then process the search based on criteria given by the Customer 
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Agent. If search fails, it returns Failed. Otherwise it returns Inform-done and Inform- 
result. 

There are other scenarios like Resource Agent registering to Directory agent, Fa- 
cilitator Agent search in the Directory for suitable Resource Agents, order negotia- 
tion between Customer Agent and Facilitator Agent, and order negotiation between 
Facilitator Agent and Resource Agent. For each scenario, an agent interaction pro- 
tocol is going to be defined. 



3.3 Multi-criteria Selection Integration 

The VCIM system purpose is to automatically decompose the customer's order into 
sub-orders and find best supplier and best paths for parts transport and assembly 
through computer based resource scheduling. This can be discomposed as two ma- 
jor processes. Process 1 is for Customer Agent to find the best Facilitator Agent. 
Process 2 is for Facilitator Agent to find the best production schedules by utiliz- 
ing suitable Resource Agents. Those processes are emulating a customer finding 
a suitable broker to making some products and the broker finding several suitable 
workshops to work together for that order. 



3.3.1 Factors in VCIM Resource Scheduling 

In previous VCIM approach, only delivery time and cost are considered in resource 
scheduling. While in real world procurement, a lot of other factors affects final de- 
cision making. To make VCIM concept more practical, we need think about similar 
procedures for real person in a company to compare quotation and select outsourc- 
ing vendors and factors inside the selection. 

According to Kumar [8], there are nine factors to be considered in Vendor Se- 
lection. These are price, location, flexible contract terms, cultural match, reputation, 
existing relationship, commitment to quality, scope of resources, added capability. 
In order to aim long term supplier relationship, Yao [9] proposed five criteria, cost, 
quality, project, and certification and delivery performance for the hierarchy. As- 
sessing a group of vendors and selecting one or more of them is a complex task be- 
cause various criteria must be considered on the decision-making process. Dickson 
[10] studied the importance of vendor evaluation criteria for industrial purchasing 
managers and presented 23 vendor attributes that managers consider in such an eval- 
uation, including quality, delivery, price, performance history and others. Weber et 
al. [11] concluded that quality was the most important factor, followed by delivery 
performance and price. They found that quality was of 'extreme importance', and 
delivery was of considerable importance. Hill [13] concluded that quality was an 
essential factor that qualified a corporation to compete in the marketplace, because 
vendors with unacceptable quality performance were dropped during the screening 
phase. 

With consideration of previous research on Vendor selection and VCIM, the com- 
parable factors for VCIM resource scheduling are identified as following: 
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Factors for Process 1, Customer Agent Find and Select the Best Suitable Facil- 
itator Agent for a Particular Order The Customer Agent finds suitable Facilita- 
tor Agents in the public registry (aka Directory Facilitator). Then it compares best 
suitable Facilitator Agent through a multi-criteria method. The comparable factors 
for this problem are: Credit, Friendship, Price, and Time. 

Credit: A property that is stored in the public registry and can be retrieved by 
request. It is variable and will be adjusted by performance history (rating system) 

Friendship: A property that is stored in the Customer Agent. It is variable and 
will be adjusted by performance history. 

Quality: A property that is stored in the public registry and can be retrieved by 
request. It is variable and will be adjusted by performance history (rating system) 

Price: A property that is generated by Facilitator Agent for that particular order 
Different Facilitator Agent has different profit margins and different quotation prices 
from other Resource Agents. Even same production schedule may have different 
facilitator quotation prices. 

Time: A property that is generated by Facilitator Agent for that particular order 

Factors for Process 2, when a Facilitator Agent Receives a Request from a Cus- 
tomer Agent, It Generates all Possible Production Schedules. Then It Finds 
the Best Suitable Production Schedule Through a Multi-criteria Method The 

comparable factors for this problem are: 

For Cost, Time, Quality, Friendship, and Credit, they are the same as in Process 1 . 
However there is one extra factor used. 

Delivery reliability: A property that is stored in the public registry and can be re- 
trieved by request. It is variable and will be adjusted by performance history (rating 
system). 

The rating system mentioned above is supposed to be a simple survey of trust 
and reputation systems like eBay's feedback form [14]. In this case it is performed 
by software agents rather than human. 



3.3.2 Multi-criteria Selection Process 

As multiple factors that have been identified in VCIM resource scheduling, a multi- 
criteria selection approach is to be integrated. In the recent years, AHP method 
has been widely adopted as a decision making tool for outsourcing vendor selec- 
tion problems. Many researches showed that the AHP is very effective solution to 
different kinds of Multi-criteria vendor selection, such as Manufacturing outsourc- 
ing vendor selection [15], Information system outsourcing vendor selection [16], E- 
business outsourcing vendor selection [17], and 3PL (third-party logistics provider) 
vendor selection of a 4PL (fourth-party logistics providers) system [18]. 

As shown in Fig. 5, the hierarchy construction has three layers for Resource 
Scheduling Process 1. Suitable Facilitator Agents are compared by five factors: 
Credit, Friendship, Quality, Price and Time to find out the most suitable Facilita- 
tor for this process. 
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The comparable factors in VCIM are quite similar to those factors in Vendor 
selection. And some factors we identified are fuzzy, such as quality and friendship. 
Therefore a Fuzzy AHP method is going to be used in the two VCIM resource 
scheduling processes mentioned above where multiple factors are involved. 



4 Conclusion 

VCIM is a way forward to the SMMEs in many countries, where the globaliza- 
tion has impacted dramatically on manufacturing industry. VCIM provides a fa- 
cility to share resources of partner enterprises that are geographically distributed 
and provides a competitive edge for SMMEs to be in equal footing with large or- 
ganizations that have abundance resources to meet the challenges imposed by the 
globalization. This article discusses an advanced Agent-based VCIM architecture 
and Multi-criteria Resource Scheduling Process that will help the SMMEs in global 
competition. As can be seen, future work to do is on optimizing Agent based re- 
source scheduling. This will involve designing new agent behavior, communication 
protocols, and an effective multi-criteria selection method Model. 
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Law Ontology 
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Abstract This paper presents an improvement of our approach called SCRO_II 
algorithm. SCRO_I algorithm was initially developed in order to retrieve a set of 
Supreme Court sentences. The goal of SCRO_I is to provide different law issues 
among those retrieved documents. We create a new ontology using different seman- 
tics to study their performances based on diversity measurement. The contribution of 
this new ontology is compared to the traditional one. A new procedure is embedded 
in SCRO_II algorithm to identify a set of synonyms and relations. The experiments 
were done on Thai Succession Law and Bill of Exchange Law. The experimental 
results show that SCRO_II outperforms SCRO_I algorithm in both data sets. 

Keywords Ontology ■ Retrieval ■ Supreme Court sentences ■ Thai succession law 



1 Introduction 

Decisions made by judges are very important for users in legal fields because these 
legal documents provide some legal doctrines that cannot be found in the code laws. 
Studying Thai law using only code laws is not possible to fulfill the knowledge of 
those users. The Supreme Court of Thailand had provided a search facility through 
the Deka System (http://deka2007.supremecourt.or.th/). The corpus of the current 
system consists of 100,010 legal documents. These documents are not predefined 
into categories of law domains; therefore the retrieval performance is not satisfied 
in terms of semantic diversity of law content presented in the documents. The reason 
that legal users need to obtain the variety of law issues from the retrieved document 
is that they need the complete understanding of each law area in all aspects. To tackle 
this problem, we propose to use an ontology as a knowledge representation and de- 
velop an algorithm called SCRO_I to generate a set of keywords that guarantee 
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the documents diversity [1]. However, the performance measured in term of doc- 
ument diversity obtained from SCRO_I algorithm is only 54.6%. The objective of 
this paper is to improve the performance of SCR0_1 algorithm. We propose a new 
ontology model that has an ability to enhance the retrieval performance. Another 
benefit of our proposed algorithm is that users do not need to input several key- 
words for searching since the algorithm will automatically create a set of keywords 
for the retrieval process. 

This paper is organized as follows. Section 2 describes an overview of ontology 
and our proposed ontology. Section 3 addresses the SCR0_1 and SCRO_II algo- 
rithm. Section 4 describes the evaluation details of ontology-based retrieval system. 
Finally we conclude our work with some discussion and directions of future work 
in Sect. 5. 



2 Ontology 

2.1 Related Works on Ontology 

Many domain specific ontologies were constructed in various areas such as medical 
domain [2], e-learning domain [3], legal domain [1, 4]. Ontologies were applied to 
enhance existing applications such as knowledge-based system [5], computational 
linguistics [6], etc. Benjamins et al. [7] applied ontology to improve IT support for 
judges in Spain. They used questionnaires as a tool to collect requirement from users 
in order to implement the ontology. E-commerce law ontology was constructed us- 
ing text mining technology by Kayed [8]. A set of conceptual nodes were extracted 
from e-commerce law cases. They proposed a new algorithm to reduce a number of 
links connected between nodes in order to decrease the ontology complexity. 

Legal Information retrieval research was done for French law [9]. Saravanan et al. 
did research about query expansion for Indian law in three area which are rent con- 
trol, income tax and sale tax. They compared their proposed algorithm to the key- 
word search and found that their algorithm outperformed the traditional retrieval 
algorithm [10]. The first Korean law ontology was generated using natural language 
processing technique cooperated with Korean WordNet. The dataset was collected 
from law text books and research papers about law. After that, their algorithm found 
a set of word co-occurrences and hand on ability to construct a graph that connects 
these words together [11]. 



2.2 Structure of Ontology 

We set up a structure of the Thai succession law ontology and Thai bill of exchange 
law ontology using a set of relations and a set of synonyms. Our ontology has two 
levels, a core level and an extended level ontology. 
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Fig. 1 The proposed framework of ontology is divided into core level and extended level 



The core level ontology represents the knowledge of Thai succession law [4]. 
Figure 1 shows an example of ontology. In case that user enters keyword, 'ri', 
SCR0_I1 algorithm traverses through the core level ontology and found the rela- 
tion 'ri' exists. Next, the algorithm explores all links attached to node between 'ri', 
(r*, r2, r3, ^4). The star sign (*) found in ri means that there is an extended knowl- 
edge related to ri . Therefore the algorithm explores the extended level of ri and 
found wordi, word2 and word3. Finally a set of queries (Q) are obtained as {A, ri, 
B}, {A, wordi, B}, {A, word2, B}, {A, words, B), respectively. 

In case that the keyword obtained from user is found in the node (e.g. 'A') of 
ontology, SCRO_II traverses through the ontology and generates the set of queries 
as follows: {A, ri,B}, {A, r2, B}, {A, r3,B}, {A, r4, C}, {A, ri,Si}, {A, r2, Si}, 
{A, rs. Si), {A, n, S2}, {A, r2, S2}, {A, r,, S2}, {A, n, S3}, {A, r2, S3}, {A, r3, 
S3}, respectively. 

We develop an application for a law expert to create Thai succession law ontology 
and Thai bill of exchange law ontology. First, the conceptual nodes were created 
when user enter keywords into the system. Then the links were connected when 
user enter words that represent the relationship between nodes. The synonym and 
reason relation are automatically created using ThaiLegalWordnet. 
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- Ontology traversal — 


A set of query 

Q = {q7,q2,q.i} 



Legal 
Documents 




List of documents 



3 Algorithms 

3.1 SCROJI Algorithm 

The framework of SCROJI algorithm is shown in Fig. 2 and Table 1. Given a 
keyword obtained from user, SCROJI searches through the core ontology for the 
keyword. If the word is found in the node of the ontology, the algorithm will traverse 
through the link to the next node. The words occurred in the nodes and relations are 
generated to form a set of query. 

In case that user enters keyword, 'A', SCROJI algorithm traverses through the 
ontology and found the node 'A'. Next, the algorithm explores all links attached 
with node 'A', (ri, r2, rs, r4, wordi, word2, word3). Then a set of queries (Q) are 
obtained as {A, ri, B}, {A, word;, B}, {A, word2, B}, {A, words, B}, {A, r2, B}, 
{A, r3, B} and {A, r4, C}, respectively. SCROJI also explores the extended level 
of ontology under two different conditions. In case that the keyword is found in the 
node, the extended level ontology will be explored only when the star sign is found 
in the node. 

On the other hand, if the keyword is the relation found in the ontology, the ex- 
tended ontology will be explored when the star sign is found on that relation. Then 
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Table 1 SCROJI algorithm 
Algorithm: SCRO II 

Input: Ontology, keyword 
Begin 

Word = { } 
Query = { } 

ConceptType = {NODE, RELATION} 
SearchConcept (Ontology, keyword) 
Word = OntologyTraversal (keyword) 
For Each Wi in Word Do 
If ConceptType (keyword) = NODE and 
ConceptType (wi ) = NODE 
If FoundExtendedOntology (wi ) 

ExtendedWord = ExtendedOntologyTraversal ( ) 
If ConcepType (wi ) = RELATION and 

ConceptType (wi ) = RELATION 
If FoundExtendedOntology (wi ) 
ExtendedWord = ExtendedOntologyTraversal ( ) 
End For 

Query = QueryGenerate (Word, ExtendWord) 
For each qi in Query 
Di = DocumentRetrieval (qi) 
Ranking (Di ) 
End For 

CombineResults = checkDuplicate (D) 
End 
Output : CombineResults 



the set of queries are created and used in the retrieval process. Considering a set 
of retrieved documents, SCRO_II algorithm ranks these documents using TFIDF 
formula (see Eq. 1). 

"^in = '/m X idf (1) 

where w\„ — weight value of wordi in the retrieved documentn. (fj^ — the number 
of occurrences for wordi in documentn. 

idf^\n(^\ (2) 

where N — the number of retrieved documents. «,- = the number of retrieved docu- 
ments contain wordi. 
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word], 
_, '',.word2,^~-.. 




SCRO_II algorithm selects the first ranked document as an output for each query 
and checks for duplicated documents. Since, each query (qi) is processed indepen- 
dently; therefore the duplicate checking process is done to finalize the output. Be- 
cause the objective of SCRO_II algorithm is to ensure the diversity of law issues 
from the final set of retrieved documents. 



3.2 SCROJ Algorithm 

The ontology used in SCROJ algorithm has only one level (Fig. 3; Table 2). Given 
a keyword, the algorithm explores through the ontology and generates a set of query. 
The set of query obtained from SCROJ is different from SCRO JI that yields dif- 
ferent performance measure in term of diversity. 

In case that the keyword obtained from user is found in the node 'A', SCROJI 
traverses through the ontology and generates the set of queries as follows: {A, ri, 
B}, {A, r2, B}, {A, r3, B), {A, r4, C), {A, wordi, B}, {A, word2, B}, {A, worda, 
B}, respectively. 

If user searches for the keyword, ri , the set of queries obtained from SCROJ are 
{A, ri, B}, {A, V2, B), {A, T3, B}, {A, wordi, B}, {A, word2, B}, {A, words, B}, 
respectively. 



3.3 Baseline Algorithm andDeka System 



We compare the performance of SCROJI algorithm with the baseline algorithm 
and Deka system. The baseline system uses term frequency and inverse document 
frequency to determine the retrieved document. Whereas the Deka system (Fig. 4) 
simply uses SQL statements to retrieve legal documents containing a keyword. The 
current system implemented by the Thai Supreme Court can be accessed via the 
website www.deka2007.supremecourt.or.th. 
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Table 2 SCROJ algorithm 
Algorithm: SCRO I 

Input: Ontology, keyword 
Begin 

Word = { } 

Query = { } 

Word = OntologyTraversal (keyword) 

Query = QueryGenerate (Word, 0) 

For each qi in Query Do 
Di = DocumentRetrieval (qi) 
Ranking (Di ) 

End For 

CombineResults = checkDuplicate (D) 
End 
Output : CombineResults 



"stititj ?i u Piti pfn w WO nw n Lia s nn a-^ Fin ^9 -3 pf 1 a g nn Mia 

;i^ uuudnsi^FiiiundnDts ^ 'BomuDuitE r'^iunns»a^«muie(u /Information 




mart itiutn-au^i i uacidNnuunil'Siiuiii 



fluu'i^'inuu'iuiavctii4iv'>nu^/ctnS4«4n«a4 






I ,ILlUf. .vHb. ] .Uniill. { .fllllBlU. 



Fig. 4 The supreme court sentences retrieval system (Deka) 

4 Experiments 
4.1 Data Set 

We collect all judge sentences related to Thai succession law and Thai bill of ex- 
change law. There are 248 and 200 documents, respectively, in the corpus. The word 
segmentation is performed since Thai language has no explicit word boundary. 
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*T- I (illegitimate 
(settleX child) 



Fig. 5 Thai succession law ontology 



The Thai succession law ontology is shown in Fig. 5 and the Thai bill of exchange 
law ontology is shown in Fig. 6. 



4.2 Performance Measurement 



Given a keyword, a set of retrieved documents are measured based on the degree of 
diversity coverage (DC). The maximum degree of diversity is 1. It means that the 
algorithm can retrieve all different law issues related to the keyword. 



DC- 



No. of law issues from the set of retrieved documents 
Total no. of law issues found in ontology 



(3) 



The retrieval performance of SCRO_II to SCRO_I, baseline algorithm and Deka 
system are compared between different types of query. 
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(aval) 





lu (receive). 
liiu (transfer). 
■iinrma'] (endorse) 
■ilfinimi (collect). 



( (indorse ) [(obi 
oncnt) 



Bf/nnlo (interest) 
/ 
ain* (draw) 

(perfori 
/ 
tiitiiio (signature) 

lou (transfer)- 
mmvti (interest) 



mlv (payment) 

ilgmi (refuse) 
\ 
ami* (draw)- 




(information) 

\ (guarantee) 

(recourse) 



Fig. 6 Thai bill of exchange law ontology 



oiiifmu (prescription) 
auauEno (bankrupt) 
infja^- 



■Bii: (perform)- 
ona*(aval) 





(pay for) 



In order to evaluate the retrieval performance on Thai succession law, we set 
up the experiment using 12 keywords which are jj^gin (estate), w^eimtu'jein (ad- 
ministrator of an estate), vntnvi (heir), wsnu (deceased), Viuiu (witness), ■wutinttu 

(wills), i^-THti (creditor), rin^eiflt'MfiJJJTiafi (exclusion for the succession), plssjJ5(>in 
(renunciation of an estate), 'WilEJnttjjljJsiJJiJ'Sfu (incomplete will), 'ditiiViU (perform- 
ing loan), 6if\3W^f^m?}J^i>in (appointment of the administrator of an estate), respec- 
tively. 

In case of Thai bill of exchange Law, we set up the experiment using 12 keywords 
which are I'Sfi (cheque), dfiufifiwu (bills of exchange), efi^fUfunt^iwu (promis- 
sory notes), wvi?o (holder), sunRn? (banker), vili (obligations), 5lsrnvisf>3 (endorse), 

iFianiUEi (interest), lau (transfer), latflEj (recourse), aTi?! (aval), ao^nti (draw), re- 
spectively. 



4.3 Experimental Results 

Considering Tables 3 and 4, we found that SCROJI can retrieve a set of legal doc- 
uments with more diversity than those of SCRO_I. In Table 3, the most difficult 
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Table 3 The comparison of performance in term of the number of law issues found by algorithms 
using sets of query on Thai succession law data set 



Keyword 


Number 
of law 


Number of law 


issues found 


by algorithms 


















SCROJl 


SCROJ 


Baseline 


Deka 




issues 










wran (estate) 


19 


14 


8 


8 


10 


w^eim^teifi 


19 


10 


10 


6 


6 


(administrator of an 












estate) 












VnEnVI (heir) 


17 


14 


9 


9 


8 


CJ'EnEI (dead person) 


4 


4 


3 


1 


2 


WEjnU (witness) 


3 


3 


3 


2 


1 


■WUtlfl^TO (wills) 


9 


6 


9 


2 


2 


mVii (creditor) 


7 


4 


3 


1 


2 


rin^sifltvffijjjtein 


6 


4 


2 


2 


6 


(exclusion for the 












succession) 












c^c^UJJTFin (renunciation 


3 


2 


2 


1 


3 


of an estate) 












■wutinffjjliipijjijfat 


4 


3 


2 


2 


1 


(incomplete will) 












lin?S;Vi'U (performing 


3 


2 


1 


1 


1 


loan) 












^ow^gimT^j^ian 


5 


3 


3 


2 


1 


(appointment of the 












administrator of an 












estate) 













keyword for algorithms are "estate" and "heir" because they serve as a general term 
of the domain, so retrieving a set of diversified documents for such keyword is con- 
sidered to be hard. For the keyword, "estate", SCROJI is able to retrieve a set of 
documents that have 14 issues whereas SCROJ gets only 8 issues. For the keyword, 
"heir", SCROJI gets 14 different issues from a set of retrieved documents whereas 
SCROJ gets 9 issues. 

In Table 4, the most difficult keyword for algorithms is "holder" because they 
serve as a general term of the domain, so retrieving a set of diversified documents 
for such keyword is considered to be hard. For the keyword, "holder", SCROJI is 
able to retrieve a set of documents that have 8 issues whereas SCROJ gets only 4 
issues. 

SCROJI and SCROJ can completely retrieve all issues related to the keyword, 
min (dead person) and viuiw (witness). Moreover, the performance of SCROJI 
and SCROJ on keyword, w'^einTjjJTOin (administrator of an estate) and efpisu'sein 
(renunciation of an estate) are the same. 

Table 5 provides details about the experiments to see the performance of algo- 
rithms using different types of query, i.e. Query _A and Query _B. Query _A contain 
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Table 4 The comparison of performance in term of the number of law issues found by algorithms 
using sets of query on Thai bill of exchange data set 



Keyword 



Number 



Number of law issues found by algorithms 





or law 
issues 


SCROJI 


SCROJ 


Baseline 


Deka 


l^Fl (cheque) 


13 


9 


10 


2 


7 


df-jueinw'u (bills of 


5 


4 


3 


1 


3 


exchange) 












^'Tsfajojntmou 


9 


6 


7 


3 


7 


(promissory notes) 












WT/ltO (holder) 


12 


8 


4 


3 


4 


fiUnfint (banker) 


5 


4 


5 


2 


4 


VI vi (obligations) 


13 


9 


10 


6 


5 


SlanViS'o (endorse) 


3 


3 


2 


2 


2 


eianilitl (interest) 


2 


2 


2 




1 


law (transfer) 


3 


1 


1 




1 


IsiltiEI (recourse) 


2 


2 


1 




1 


aTJc) (aval) 


6 


5 


4 




3 


^v3^T£J (draw) 


2 


2 


1 




1 



Table 5 Comparison between types of query in each algorithm 



Algorithm 



Query_A 



Query_B 





Succession 


Bill of exchange 


Succession 


Bill of exchange 




Law 


Law 


Law 


Law 


SCROJI 


0.34 


0.33 


0.72 


0.79 


SCROJ 


0.35 


0.37 


0.59 


0.66 


Baseline 


0.30 


0.32 


0.37 


0.37 


Deka 


0.34 


0.25 


0.45 


0.26 



all keywords getting from ExtendedOntologyTraversal ( ) in SCRO_II al- 
gorithm. On the other hand, Query _B are set of query obtain from QueryGener- 
ate(). 

The performance measured on diversity of documents shows that the average per- 
formance of SCRO_II is higher than that of SCROJ. SCRO_II gets 72%, whereas 
SCROJ gets 59%. Because the two-level ontology is able to expand sets of query 
more than the ontology of SCROJ which has only one level. 

Consider Thai succession law data set, the performance measured on diversity of 
documents using Query _A, SCROJl and SCROJ get 34% and 35% respectively, 
whereas the performance of SCROJl and SCROJ using Query _B get 72% and 
59% respectively. For Thai bill of exchange Law data set, the performance mea- 
sured on diversity of documents of Query _A, SCROJl and SCROJ get 33% and 
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Fig. 7 Query expansion 
process in SCROJI 



ExtendedOntologyTraversal {) 



Query_A 



QueryGenerate ( ) 



Query_ 



DocumentRetrieval ( ) 



Sets of document 



37% respectively, where as the performance measured on diversity of documents of 
Query_B, SCRO_II and SCROJ get 79% and 66%, respectively. 

Hence, using Query _B to retrieve documents can obtain diversify documents for 
each issue but Query _A has less performance in retrieving such diversify documents 
because a single set does not represent each issues. 

Consider ontology shown in Fig. 1, if user enter keyword 'A' then Extended- 
OntologyTraversal ( ) returns a query (name as Query_A) {ri, r2, rs, r4, 
wordi, wordi, words}. On the other hand, QueryGenerate ( ) returns sets of 
query (Query_B) {A, ri, B}, {A, wordi, B}, {A, word2, B}, {A, words, B), {A, r2, 
B}, {A, rs, B) and {A, r4, C}, respectively. 

Consider Fig. 7, QueryGenerate ( ) plays an important role in the docu- 
ment retrieval process. Using Query _A in Thai Succession law data set, SCRO_II, 
SCROJ, Baseline and Deka system get 34%, 35%, 30% and 34% respectively. 
Whereas SCRO_II, SCROJ, Baseline and Deka system using Query _B get 72%, 
66%, 37% and 26% respectively. Because each subset of Query_B is considered a 
law issue, but Query _A cannot exhibits the law issue diversity. 

Considering the baseline algorithm, we surprisingly found that the Deka system 
using SQL statement outperforms the traditional information retrieval system using 
TFIDF formula. The reason is that Deka system simply retrieves judges sentences 
having the keyword and sorts by the document ID no. It is assumed that the succes- 
sion law cases submitted to the Supreme Court are likely to have different issues. 
However, the Deka system has less diversity value than SCRO_II because the crite- 
ria to determine the performance of any information retrieval system concerns about 
the ranking process. We want the system to deliver the suitable number of choices 
(documents) to users so that they can effectively view their search results. There- 
fore, the objective of our research is to develop an algorithm to effectively retrieve 
a suitable number of judge sentences that guarantee the variety of law issues. 
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5 Conclusions 

This study proposes an ontological approach for Supreme Court Sentence Retrieval 
on Thai succession law data set and Thai bill of exchange law dataset. The objective 
of using ontology is to provide a set of diversify documents for user. We found that 
using hierarchical style of ontology can provide more variety of law issues in a set 
of retrieval results. The experimental results demonstrate that SCRO_II outperforms 
SCRO_I algorithm in term of diversity value because ontology structure can expand 
sets of query which have a diversity of law issues. 

Comparing between type of query, i.e. Query _A and Query _B, we found that sets 
of query exhibit diversity results more than a set of query. Beside, users are satisfied 
with the judge documents retrieved by SCRO_II. 
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School of Kasetsart University. 
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Genetic Algorithm Based Model for Effective 
Document Retrieval 



Hazra Imran and Aditi Sharan 



Abstract One central problem of information retrieval is to determine the rele- 
vance of documents with respect to the user information needs. The choice of sim- 
ilarity measure is crucial for improving search effectiveness of a retrieval system. 
Different similarity measures have been suggested to match the query and docu- 
ments. Some of the popular measures being: cosine, jaccard, dice, okapi etc., each 
having their own pros and cons. Accordingly one may give better result over other 
depending on users need, document corpus, organization and indexing of corpus. 
Therefore it may be justifiable to combine these measures and develop a new simi- 
larity measure which can be named as combined similarity measure. Now individ- 
ual measures can be assigned weights in different proportion in combined similarity 
measure. In order to optimize ranking of relevant documents, individual weights 
have to be optimized. In this chapter we suggest a genetic algorithm based model 
for learning weights of individual components of combined similarity measure. We 
have considered two different types of functions viz: non-order based and order 
based fitness functions to evaluate the goodness of the solution. A non-order based 
fitness function is based on recall-precision values only. However, it has been ob- 
served that a better fitness function can be obtained if we also consider the order in 
which relevant documents are retrieved. This leads to an idea of order based fitness 
functions. We evaluated the efficacy of a genetic algorithm with various fitness func- 
tions. The experiments have been carried out on TREC data collection. The results 
have been compared with various well-known similarity measures. 

Keywords Document ranking ■ Genetic algorithms ■ Similarity measures ■ 
Information retrieval ■ Vector space model 



H. Imran (El) 

Department of Computer Science, Jamia Hamdard, Hamdard Nagar, New Delhi 110 062, India 

e-mail: himran@jamiahamdard.ac.in 

S.-I. Ao et al. (eds.), Intelligent Control and Computer Engineering, 191 

Lecture Notes in Electrical Engineering 70, 

DOI 10.1007/978-94-007-0286-8_16, ©Springer Science+Business Media B.V. 2011 



192 H. Imran and A. Sharan 

1 Introduction 

An Information Retrieval (IR) is one that accepts a user query and returns a ranked 
list of relevant documents based on users query. The efficiency of IR system is en- 
tirely dependent on similarity measure used for ranking the documents. Thus choice 
of appropriate similarity measure is most crucial aspect for developing an efficient 
IR system. Different similarity measures have been suggested for matching query 
and document, each having their own pros and cons. Sometimes it may be justifi- 
able to combine various similarity measures leading to a new similarity measure, 
which can be called as combined similarity measure (CSM). Individual measures 
in CSM can be given weight in different proportions. In order to obtain optimal 
ranking weights of these individual measures need to be optimized. In this chapter 
we suggest a GA based model for efficient matching of query and documents us- 
ing combined similarity measure. The chapter is divided in 6 sections. In Sect. 2 
we briefly introduce IR related terminology, which includes Vector Space Model 
(VSM), similarity measures, and measures for evaluating IR system. In Sect. 3 we 
introduce genetic algorithm and its functioning. Section 4 deals with our proposed 
model. In Sect. 5 we present experiments and results. Finally Sect. 6 concludes our 
work. 



2 Information Retrieval System 

Information retrieval system is devoted to finding "relevant" documents with re- 
spect to users query. Standard IR systems are based on Boolean [1], vector [6], and 
probabilistic models [8]. Each model describes documents, queries and provides al- 
gorithms to compute similarity between user's query and documents. The objective 
of an information retrieval system is to provide its users with satisfactory retrieval 
results. 



2.1 Vector Space Model 

Our proposed model is based on Vector Space Model (VSM). In VSM, both doc- 
uments and the user given query are represented as vector of terms. Suppose there 
are t index terms in a collection of documents. Then document Z), and query Q can 
be represented as 

Di^(dn,di2,-.-,di,) (1) 

Q^{Wql,Wq2,...,Wqt) (2) 

where dij {j — \iot) are term weights in document Z),- and Wqj {j —IXot) are term 
weights in the query Q. There are different methods of assigning weights to dij. 
Representative weighting functions include term frequency (TF), inverse document 
frequency (IDF), the product of TF and IDF (TFIDF). 



Genetic Algorithm Based Model for Effective Document Retrieval 193 

2.1.1 Term Frequency 

The TF method assumes that high frequency content bearing terms represent the 
main topics of the document, and measures the frequency of occurrence of an index 
term in the document text as its weight, that is, 

TFij — Frequency of yth term in document i (3) 



2.1.2 Inverse Document Frequency 

However, terms may occur frequently in a document yet be poor indicators of its 
content. For example, the term computer is not a good index term for a document 
collection in computing. The fewer the texts in which a term appears, the more 
discriminating that term is. Therefore, the weight of a term should be inversely 
related to the number of documents in which the term appears. The IDF defined by 
Spark Jones [10] is commonly used to incorporate this effect; it is computed as 

\ogN 
idfj = ^— (4) 

rij 

where A^ is the number of documents in the reference collection and rij is the num- 
ber of documents in the reference collection having index term j . Within a docu- 
ment collection, the best terms for a particular document will be those occurring 
frequently in that document but rarely in the other documents. 



2.1.3 Term Frequency-Inverse Document Frequency Weight 

The above findings form the basis for a very popular term weighting function that 
determines the product of the TF and the IDF (TF ■ IDF) [7, 9] of the index term: 

^'./ = TFij X idfj (5) 

It captures the essence of two main assumptions: 

1. A word with high term frequency (i.e. that occurs many times in the document) 
is more likely to describe the contents of the document than a word that only 
occurs one or a few times, and 

2. A word with high document frequency (i.e. that occurs in many documents) is 
less likely to discriminate between relevant and non-relevant documents than a 
word that only occurs in a few documents. 



2.2 Ranking of Documents 

Once documents and query are arranged in VSM, a vector based similarity measure 
can be used for matching query and document. For calculating similarity of query 
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Fig. 1 Precision and Recall Document Collection 
for a given example 
information request 




Relevance Judgement Retrieved Documents 

Retrieved and Relevant Documents 

and document cosine, jaccard and dice are the popularly used similarity measures. 
Cosine, jaccard and dice similarity measures are defined as follows: 

CosiQ,Di)^- ^'-^ '■' ■'- (6) 

As the angle between the vectors shortens, the cosine angle approaches 1, meaning 
that the two vectors are getting closer, meaning that the vectors represent the sim- 
ilarity of document and query increases. The Jaccard coefficient is defined as the 
size of the intersection divided by the size of the union of the document and query 
vectors. 

Jaccard( 0,0,)^ —, / -, (7) 

For document and query vector, the dice coefficient may be defined as twice the 
shared information (intersection) over the combined set (union) 

Dice(e, A) = ^ (8) 



Ei=iW./)'E^=i('^../)' 



23 Evaluation 

Recall and precision are basic evaluation parameters (Fig. 1). For an example infor- 
mation request /, a set of relevant documents R is given, with \R\ being the number 
of documents in this set. The evaluated retrieval strategy yields the document set A 
for the example request, likewise \A\ being the number of documents in the set. Let 
\Ra\ be the number of documents in the intersection of these two sets. 

Definition Recall indicates the relevant documents, which have been retrieved. 

Recall = ^—^ (9) 

1^1 



Definition Precision indicates how many retrieved documents are relevant. 

Precision = (10) 

\A\ 
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Recall and precision evaluate IR system along different dimensions. Precision 
measures the accuracy of the result in topmost ranked documents whereas recall 
measures quality of overall answer and breadth of algorithm. In many situations, 
the use of single measure which combines recall and precision could be more ap- 
propriate. One such measure is F measure 

2 * Recall * Precision 

F-measure= (11) 

(Recall + Precision) 

This is also known as the F\ measure, because recall and precision are evenly 
weighted. 



3 Genetic Algorithms 

The genetic algorithms represent an artificial intelligence search technique that emu- 
lates the process of the natural selection. As input they have a population of individ- 
uals known as chromosomes, which represent the possible solutions to the problem. 
Initially these inputs are randomly generated, although if there is some knowledge 
available concerning the said problem, it can be used to create part of the initial set 
of potential solutions [4]. These individuals evolve in successive iterations known 
as generations, by means of processes of selection, crossover, and mutation. These 
iterations halt when the system no longer improves, or when a preset maximum 
number of generations is reached. The output of the GA will be the best individual 
of the end population, or a combination of the best chromosomes of that population. 
GA requires the evaluations of the fitness function to assign a utility value to every 
solution produced. 

Genetic Algorithms are learning algorithms as well as offer a domain indepen- 
dent search ability that can be used in many learning tasks. Due to this reason, the 
application of GAs to IR has increased in the last decade [3, 5]. 



4 Our Approach 

In our problem, we have defined a combined similarity measure [2] based on genetic 
algorithm consisting of standard similarity measures. 



4.1 Combined Similarity Measure Using Genetic Algorithm 

Our combined similarity measure is a weighted sum of the scores returned by dif- 
ferent matching measures. 

In general a combined similarity measure is represented as: 

M 

combined _similarity _measure{D , Q)—y (vvf,- x SMi(D, Q)) (12) 

(=1 
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where SMi {D, Q) signifies the score of document D for given query Q for ith sim- 
ilarity measure. M is total number of standard similarity measures considered. We 
have used three similarity measures: cos{SMi) and jaccard(5M2) and dice(5'M3). 
wtj is the weight assigned to ith similarity measure. Weights wt\ and wt2 range 
from 0.0 to 1.0. A less weight signifies that associated similarity measure is less 
significant. We assign appropriate weights to the components of these measures so 
as to achieve maximum retrieval efficiency. 

In next subsection we present our genetic algorithm based model for finding 
optimal weights using combined similarity measure (Fig. 2). 



4.2 Proposed Model 



Figure 4 presents our proposed model. The input to the model is document and 
query and output is the optimized weights for combined similarity measure. The 
working of module is as below: 

1. Documents and queries are preprocessed and tokenized. During preprocessing 
we perform stop word removal and stemming. The result obtained is a collection 
of tokens. 

2. Documents and queries are indexed using Vector Space Model. We assigned 
weights to the terms in each document by the classical tf.idf scheme. 

3. Documents are matched with queries with different similarity measures. We have 
used similarity measures: cos, jaccard and dice. These values are returned as 
SMi,SM2,...,SM„ 
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Fig. 4 Our proposed genetic algorithm based model 



4. The population module generates chromosomes for initial population of genetic 
algorithm. The chromosome is represented in the following way (see Fig. 3), 
where wt, — weight of the /th similarity measure. We have used a real- valued 
chromosome because it is more natural representation for our problem and it 
decreases dramatically the number of genes required to specify a design, thus 
making the solution space easier to search. 

5. Considering similarity measures and weights as input, combined similarity mod- 
ule calculates similarity of the document with respect to combined similarity 
measure using Eq. 12. 

6. Fitness module is used to find the fitness of the solution. Fitness function is a 
performance measure that is used to evaluate how good each solution is. Given 
a chromosome, the fitness function must return a numerical value that represents 
the chromosome's utility. Previous work has been done in GA considering fit- 
ness functions based on recall and precision only. However it has been observed 
that a better fitness function can be obtained if we also consider the order of 
the retrieval of documents. Order-Based Fitness function takes into account the 
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number of relevant and irrelevant documents along with the order of their ap- 
pearance. Note that the documents retrieved earlier in order have higher utility 
in comparison to documents retrieved later. The basic idea being that it is not 
the same that the relevant documents appear at the beginning or at the end of the 
list of retrieved documents. We have used following non-order and order-based 
fitness functions. 

(2 X recall x precision) 
Fitness lnon-order-based= — (13) 

recall + precision 

1 ^/ ^l\ 

Fitness2order-based = 7777 > r{di) y - (14) 

where |Z>| is the total number of documents retrieved, and r{d) is the function 
that returns the relevance of document d, giving 1 if the document is relevant and 
a otherwise. The importance of the order-based fitness function can be justified 
with the following example. Let us assume that two similarity measures retrieve 
6 relevant documents in the top 15 result giving Case 1 and Case 2, where 

Casein 100100100111000 
Case 2= 111010110000000 

The relevance information is coded as 1 (if relevant) and (if irrelevant). Here 
we observe that the recall and precision in both the cases is same. However one 
can easily see that Case 2 has a better performance as all the 6 documents are 
retrieved earlier in comparison to Case 1. If we consider the total number of 
relevant document as 10 then the values for Fitness 1 non-order-based is .48 for both 
the cases whereas Fitness2oi-der-based are calculated as 0.45 for Casel and 0.64 for 
Case2. Therefore we can justify that Fitness2order-based is better. 

The fitness module sorts the population on the basis of the combined similar- 
ity measure obtained. Further precision and recall are calculated and finally the 
fitness is calculated for each member of the population using Eqs. 7 and 8. Once 
the fitness is calculated next modules applies standard GA functions: selection, 
crossover and mutation and generates new population. The whole process is re- 
peated iteratively until population converges or maximum number of iterations 
has been carried out. 
7. Genetic Algorithm module applies the standard GA functions (selection, 
crossover, mutation) to generate new population from the old population, which 
is discussed as follows. 

• Selection: - Selection embodies the principle of 'survival of the fittest'. Chro- 
mosomes having higher fitness are selected for crossover. The roulette wheel 
reproduction process [1 1] was used to select individuals. 

• Crossover: - Crossover is the genetic operator that combines two chromo- 
somes together to form new chromosome. We have used single point crossover 
where a locus position is selected within two parent chromosomes and the 
genes are swapped from that position to the end of parent. 
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Fig. 5 Curve showing 
variation of average fitness 
with generation number for 
both the fitness function 
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• Mutation: - Mutation involves the modification of the values of each gene of 
a solution with some probability (mutation probability). 



5 Experiments and Results 

For our experiments, we used volume 1 of the TIPSTER document collection, a 
standard test collection in the IR community. Volume 1 is a 1.2 Gbyte collection of 
full-text articles and abstracts, divided into seven main files. The documents came 
from the following sources. 

1. WSJ -Wall Street Journal (1986, 1987, 1988, 1989, 1990, 1991 and 1992) 

2. AP - AP Newswire (1988, 1989 and 1990) 

3. ZIFF - Information from Computer Select disks (Ziff-Davis Publishing) 

4. FR- Federal Register (1988) 

5. DOE - Short abstracts from Department of Energy 

The experiments were done for non-order-based (Fitness Inonorderbased) and or- 
der based (Fitness2orderbased) fitness functions. The control parameters set empir- 
ically were as follows: population size — 100, probability of crossover (Pc — 0.7), 
and probability of mutation (Pm — 0.01). The document cut off was 10. We have 
performed our experiment on 50 queries. Below is the example of applying Genetic 
Algorithm based model on TREC query 51 [Airbus Subsidies]. 

Individual [0.862747, 0.064331, 0.656781] Fitness: 0.602898 

Individual [0.862747, 0.064331, .6567812] Fitness: 0.605797 

Individual [0.862747, 0.064331, 0.656781] Fitness: 0.608695 

Individual [0.702370, 0.237541, 0.921758] Fitness: 0.602898 



Below are the graphs to show the results of experiment. 
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Fig. 6 Curve showing recall 
and precision graph of cos, 
Jaccard, Dice and combined 
similarity measure 
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Figure 5 shows variation of average fitness with generation number for a specific 
query 5 1 of TREC dataset. As shown the average fitness increases with generation 
number. On the basis of our experiment we observed that for 55% and 60% of the 
queries of TREC dataset has increased average fitness in successive generations. 
We compared the results of our experiments with the recall and precision values ob- 
tained by all three standard similarity measure i.e. Cos, Jaccard Dice and combined 
similarity measure. For almost all the queries our experiment gave better results. 
Figure 6 shows recall precision curve for all the three standard measures and CSM 
for a specific query in TREC. Document cutoff was 10. Recall-precision curve is a 
standard curve for measuring efficiency of any retrieval algorithm. They are useful 
because they allow us to evaluate quantitatively both quality answer and breadth of 
retrieval algorithm. Good quality information retrieval algorithm gives high preci- 
sion at low recall values. One can easily observe in Fig. 6 that initially all similarity 
measures start with reasonable precision values (between 0.4 to 0.5). However stan- 
dard similarity measures show a sharp decline in the beginning whereas our com- 
bined similarity measure shows gradual decrease in precision. This indicates that 
combined similarity measure gives better quality of result. The breadth of combined 
similarity measure is maximum as it is giving a recall value 1 . 



6 Conclusion 



In this paper we have developed a GA based model for efficient matching of query 
and documents by an Information Retrieval System. An Information Retrieval sys- 
tem may use different similarity measures for matching query and document. Each 
of these measures have their own pros and cons. Choice of appropriate similarity 
measure is most crucial aspect that determines efficiency of an IR system. In this 
work we suggest use of combined similarity measure, which combines individual 
similarity measures giving them appropriate weights in order to obtain optimally 
ranked document set. GA has been used for learning these optimum weights. We 
have used two different type of fitness functions viz: non-order and order based fit- 
ness function. Extensive experiments have been performed on TREC data set. As 
expected GA improves efficiency of IRS, order based fitness being more efficient. 
This work provides a framework for exploring use of Genetic algorithms to improve 
efficiency of an Information Retrieval System. 
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Abstract This paper presents a Cloud service discovery system (CSDS) that 
aims to support Cloud users in finding a Cloud service over the Internet. The 
CSDS interacts with a Cloud ontology to determine the similarities among ser- 
vices. The significance of this project is that it is the first attempt in building an 
agent-based discovery system that consults an ontology when retrieving informa- 
tion about Cloud services. One of the main contributions of this work is build- 
ing a Cloud service reasoning agent (CSRA) that enables the CSDS to (1) reason 
about the relations of Cloud services and (2) rate the search results. Another con- 
tribution of this work is designing and constructing a Cloud ontology consisting 
of a taxonomy of concepts of Cloud services that enables the CSRA to determine 
the relations of Cloud services using three service reasoning methods: (1) simi- 
larity reasoning, (2) equivalent reasoning, and (3) numerical reasoning. Empirical 
results show that using the Cloud ontology, the CSDS is more successful in find- 
ing Cloud services that are closer to users' requirements. Experiments are also 
conducted to examine the effect of using different combinations of the three ser- 
vice reasoning method: (1) using only similarity reasoning, (2) using similarity rea- 
soning and equivalent reasoning, and (3) using all three reasoning methods. Addi- 
tionally, a proof-of-concept example demonstrates the major functionalities of the 
CSDS. 
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1 Introduction 

Cloud computing is Internet (Cloud) based development and the use of computer 
technology (computing) whereby dynamically scalable and often virtualized re- 
sources are provided as a service over the Internet [1]. Consumers of Cloud comput- 
ing will not compute using their own computer, but move their programs and data 
to the Clouds consisting of computation and storage utilities provided by third par- 
ties. Cloud computing providers publish Cloud services over the Internet, and con- 
sumers normally access these services provided by Cloud application layer through 
web-portals [2]. 

To date, however, there is no discovery mechanism for searching different kinds 
of Clouds. Cloud consumers generally have to search for appropriate Cloud ser- 
vices manually [3]. Even though there are many existing generic search engines 
that consumers can use for finding Cloud services, these engines may return URLs 
containing not relevant web-pages to meet the original service requirements of con- 
sumers. Intuitively, visiting all the web-pages can be a time-consuming job. Whereas 
generic search engines (e.g., Google, MSN, etc.) are very effective tools for search- 
ing URLs for generic user queries, they are not designed to reason about the rela- 
tions among the different types of Cloud services and determining which service(s) 
would be the best or most appropriate service for meeting consumers' service re- 
quirements. Hence, service discovery mechanisms for reasoning about similarity 
relations among Cloud services are needed. The significance of this work is that to 
the best of the authors' knowledge, it is the earliest effort in constructing a Cloud 
service discovery system (CSDS) to assist users in searching for Cloud services 
more efficiently. However, it is noted here that this work is not designed to compete 
with or to replace existing generic search engines. Rather, the CSDS in this work 
employs existing search engines as its initial searching mechanism for gathering 
information about the websites of Cloud services. Then, by consulting a Cloud on- 
tology, the CSDS attempts to recognize an appropriate Cloud service among a list 
of several services. When a consumer submit requests to find Cloud services with 
specific requirements, the CSDS returns the best service and recommends other ser- 
vices for the user. 

The objectives of this project are (1) to develop a CSDS (Sect. 2), (2) to design 
and construct a Cloud ontology (Sect. 4), and (3) a Cloud service reasoning agent 
(Sect. 3) for reasoning about the relations among Cloud concepts by consulting the 
Cloud ontology. 



2 A Cloud Service Discovery System 

This section illustrates the prototype of a Cloud service discovery system (CSDS) 
consisting of a search engine and three different agents, query processing agent, fil- 
tering agent, and Cloud service reasoning agent (CSRA). In Fig. 1, there are two 
main components, (1) a CSDS, which helps to find the best Cloud service in be- 
half of users, and (2) a Cloud ontology, which consists of taxonomy of concepts of 
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Fig. 1 Cloud service discovery system 



different Cloud services to consult with the CSRA. In addition, there is a user inter- 
face that allows the user to enter queries containing a service name and requirements 
considered by their preferences. 

Query Processing Agent (QPA). The QPA locates information sources by exe- 
cuting conventional search engines. Although the selection of a search engine is ar- 
bitrary, the default search engine is Google (Search API). If the number of searched 
results is fewer than that specified by a user, generate new alternate queries to have 
more results [4]. The QPA works as follows: 

1 . Search with an exact service name (QPA refers to the Cloud ontology) 

2. If the number of searched results < 50 then. 

Generate new alternate queries used by its sibling nodes in Cloud ontology 
// With a different service name, but the service provides same (similar) 
functionalities 



Filtering Agent. It is to relieve users of time consuming and laborious tasks 
of surfing many websites during an information retrieval process [4]. The filtering 
agent removes web-pages, (1) which cannot open URLs (not available) and (2) that 
sum of distance is high between keywords of users and a web-page by calculating 
distance of each term in the Cloud ontology. 
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Cloud Service Reasoning Agent (CSRA). It consults the Cloud ontology to 
reason about the relations among Cloud services. There are three reasoning methods 
to determine the similarity among services. Details of functionalities of the CSRA 
are given in Sect. 3. 



3 A Cloud Service Reasoning Agent 

A CSRA carries out two functions, (1) reasoning and (2) rating, as shown in Algo- 
rithm 1. 

Reasoning. Reasoning is needed to understand a service and its similarity among 
a list of services. It makes a difference of the service similarity by calculating the 
similarity between the users input and services. A CSRA consults a Cloud ontol- 
ogy for performing service reasoning. All information supplied by a user is used 
to determine the similarity between two services. There are three methods to deter- 
mine similarity, (1) similarity reasoning, (2) equivalent reasoning, and (3) numerical 
reasoning. 

Rating. An aggregated similarity of each term in a web-page (i.e., service utility) 
is used to determine the rating. A web-page that has the highest service utility would 
be selected as the best service for the user. Other recommended services would be 
selected as well. 

Algoritlim 1 

For all filtered web-pages IFt(l), Fti2), . . . , Ft{N)} 
For number of all terms in a web-page, i — 1,2,3, ... ,n 

1. Calculate similarity between q(i) in users queries {q(l),q{2), . . . ,q(n)}, 
and term f (/) in Ft(N) {t(l), t(2), ..., f («)}. 

If two terms, q{i) and t{i), are numeric values, 

(3) Numerical reasoning 
else 

(1) Similarity reasoning 

If two terms, q{i) and t{i), are sibling nodes, then 
(2) Equivalent reasoning 
endFor 

2. Aggregate the similarity of all terms as ServiceUtility in the web-page 
{t(l),t(2),...,t(n)} 

[Aggregation method] 

ServiceUtility = Ylk=i term(k) x weight(k) 
where weight(k) — \/n is uniformly distributed. 
endFor 

3. Rate a list of web-pages used by ServiceUtility in descending order. 

4. Select a web-page that has the highest ServiceUtility as the best Cloud 
service, and select other recommendation services as well. 
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4 Cloud Ontology 

An ontology can provide meta information, which describes data semantics [5]. It 
provides a shared understanding of a domain of interest to support communication 
among human and computer agents [6] . An ontology contains a set of concepts and 
relationship between concepts, and can be applied to information retrieval to deal 
with user queries [7]. 

In Cloud computing. Clouds are generally divided into three different levels 
(laaS, PaaS, and SaaS [8]). 

Infrastructure as a Service (laaS) [8] provides hardware, software, and equip- 
ments to deliver software application environments with a resource-usage-based 
pricing model. 

Platform as a Service (PaaS) [8] offers a high-level integrated environment to 
build, test, and deploy custom applications. Generally, developers will need to ac- 
cept some restrictions on the type of software they can write in exchange for built-in 
application scalability. 

Software as a Service (SaaS) [8] delivers special-purpose software that is re- 
motely accessible by consumers through the Internet with a usage-based pricing 
model. 

The Cloud ontology in this work represents the relations among Cloud services to 
facilitate the CSRA in reasoning about the relations among Cloud service concepts 
(Figs. 2 and 3). It consists of 424 concepts constructed for the service reasoning. 
These include concepts of Cloud services that are currently being used and many 
services that may be released in the near future. There are three kinds of reason- 
ing methods, (1) similarity reasoning, (2) equivalent reasoning, and (3) numerical 
reasoning. 



4.1 Similarity Reasoning 

Similarity reasoning is to calculate similarity between two concepts by count- 
ing common reachable nodes. The similarity of concepts represents the degree of 
commonality between concepts. We compute the semantic similarity based on the 
method in [10] as follows: 

. |Q'(x)na(>')| \a(x)na(y)\ 

sim(x,y)^p — — h(l-p) — — (1) 

\a(x)\ \a(y)\ 

where p e [0, 1] determines the degree of influence of generalizations. 

a(x) is the set of nodes (upwards) reachable from x, we have a(x) n a(y) as 
the reachable nodes shared by x and y, which is an indication of the commonality 
between concepts x and y [10]. 

For example, see Fig. 2 (laaS-InfraSoftware-OS). In terms of Eq. 1, the concepts 
UNIX and Windows have 4 reachable nodes (upwards) from themselves, namely, 

a(Unix) — 4, a(Windows) — 4, a(Linux) — 5, a(Unix) PI a(Windows) — 3, 
a(Unix) fl a(Linux) — 4. Then, the similarity of a(Unix) n a(Linux) = 4 is greater 
than a(Unix) n aCWindows) — 3. 
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Fig. 2 Relation in terms of OS: laaS 
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Windows 




4.2 Equivalent Reasoning 



Equivalent reasoning is used to calculate the similarity between two sibling concepts 
based on its labeled value. Since two sibling concepts have the same similarity, 
equivalent reasoning is designed to make the difference in the similarity between 
the concepts. We compute the similarity between two concepts, x and y, using the 
following method: 



sim(x, y) — sim(x, y) + 



(O.8IC1-C2I) 

To 



(2) 



where Ci is a labeled value of concept x, C2 is a labeled value of concept y, and 
simix, y) is calculated by (1) similarity reasoning. 

The values from 1 to 9 represent the chronological order of Windows (Fig. 3). 
This can be the way of classification of sibling concepts. In this way, "Windows95" 
is assigned the value 1 meaning the oldest version, and "Windows?" is assigned the 
value 9 meaning the latest version among all children nodes. Examples of showing 
the difference of the similarity between two sibling concepts are as follows. 



a) Windows? and Windows Vista 

n Q\Wmdowsl-WindowsVista\ r\ q2 



10 10 

b) Windows? and WindowsXP 

Q o\Windowsl —WindowsXP\ r\ q4 



= 0.064 



10 



10 



: 0.041 
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Table 1 Maximum and 
minimum values of each 


Concept 


Max 


Min 


concept 


CPU clock 


0.1 


5.0 




RAM 


0.256 


32 




HDD 


0.1 


10000 




Network bandwidth 


0.1 


10 




Network latency 


1 


50000 



Thus, even though two concepts are sibling nodes and could have same similarity, 
we make the even small difference between the concepts by equivalent reasoning. 



4.3 Numerical Reasoning 



Numerical reasoning is used to determine the similarity between two numeric values 
of concepts. We compute the similarity over numerical values using the following 
method. 



sim(a, b, c) — I 



Maxc — Mine 



(3) 



where a and b are two numeric values and c is a name of concept that refers to 
maximum and minimum values of each concept in the Table 1 . 

For example, the similarity between two numeric values, 3.2 and 2.6, in the con- 
cept of CPU clock is determined as follows: 



sim(3.2, 2.6, CPUClock) = 1 



3.2-2.6 



5.0-0.1 



:0. 



5 Experimentation and Evaluation 



The performance measures are (i) service utility (Fig. 4) and (ii) success rate (Fig. 6), 
with three comparison schemes, searching a Cloud service (1) without the CSDS, 
(2) the CSDS without the Cloud ontology, and (3) the CSDS with the Cloud on- 
tology. In case (1), web-pages are searched with an exact service name and a web- 
page is randomly selected from the searched results. If it is a web-page about a 
Cloud service, then its service utility is determined. If not, the service utility is as- 
signed as zero, which means that the discovery has failed. In case (2), web-pages 
that do not include a Cloud term are filtered out from the searched results, and a 
web-page is selected randomly from the filtered results, and the service utility is cal- 
culated. In case (3), web-pages are rated by the aggregated service utility, which is 
a result of the service reasoning. We conducted additional experimentation (Figs. 5 
and 7) extended from [11] with three different compositions of reasoning methods. 
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Table 2 Experiment settings for simulations 



Experiment variables 



Value (range) 



The number of providers 

The number of Cloud services provided 
by each provider 

The number of Cloud service web-pages 
in the virtual-www 

The number of web-pages in the 
virtual-www (not for Cloud service) 

Total number of web-pages in the 
virtual-www 

The number of Cloud services 

CPU clock 

RAM size 

HDD size 

Network bandwidth 

Network latency 



40,70, 100, 130, 160, 190 
25 ~ 35(web-pages) 

1200, 2100, 3000, 3900, 4800, 5700 (web-pages) 

10000 web-pages 

11200, 12100, 13000, 13900, 14800, 15700 

Around 100 service names 

0.1 ~ 6.0 GHz 

0.256 ~ 36.0 GB 

0.1 ~ 1000 GB 

0.1~10Gbps 

1 ~ 5000 ms 



Fig. 4 Service utility 



1.00 
0.80 
■B 0.60 
■| 0.40 
0.20 
0.00 

















•- 






- A-^ 




-* 
















— "■• without CSDS 






X.. 


-V' 




'■•v 


* CSDS without CO 


y-: 


-■X'' 










— •- CSDS with CO 



40 70 100 IJO 160 190 
The number of Cloud service providers 



(1) Sim. -|- Equ. -|- Num., (2) Sim. -|- Equ., and (3) Sim., to show how each reason- 
ing method affects to the performance of the CSDS and the importance of all three 
reasoning methods for the CSDS as a result. 

For evaluation purpose, we assumed that the WWW is replaced by the virtual- 
www for ease of testing. There are already 10000 web-pages (not for Cloud ser- 
vices) in a directory called the virtual-www, and around 30 web-pages are automat- 
ically generated by each provider when the CSDS is deployed. Depending on the 
number of providers generated, the total number of web-pages (i.e.. Cloud services) 
would be decided between 1 1200 and 15700 in the virtual-www. The CSDS requires 
information containing a service name, the OS, CPU name, and the value of CPU 
clock, RAM, HDD, network bandwidth, and network latency (see Table 2). 

Service Utility. In Fig. 4, the result of the CSDS with the Cloud ontology shows 
higher performance than without the CSDS and the CSDS without the Cloud on- 
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Fig. 5 Service utility: 
Reasoning methods 



Fig. 6 Success rate 



Fig. 7 Success rate: 
Reasoning methods 
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tology in terms of the service utility. Tiiis is because the CSDS has filtering and 
reasoning functionalities, which means that web-pages of the Cloud service have 
higher chance to be selected and is more likely to be closer to users' requirements. 
In the Fig. 5, the result shows that selecting either without numerical reasoning or 
without equivalent reasoning and numerical reasoning may affect the performance 
of the CSDS in terms of service utility, since five numeric values (the value of CPU 
clock, RAM, HDD, network bandwidth, and network latency) should be consid- 
ered for reasoning between users and Cloud services. The reasoning with similarity 
reasoning and equivalent reasoning methods show slightly better performance than 
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Fig. 8 After searching and filtering from the virtual-www 

with only similarity reasoning, because the equivalent reasoning was designed to 
make a difference of the similarity between two sibling concepts (two sibling con- 
cepts have the same similarity by the similarity reasoning), and the value of similar- 
ity is very small calculated by equivalent reasoning (shown in Sect. 4.2, examples 
of equivalent reasoning). 

Success Rate. In Fig. 6, success rate is calculated by the number of successes/ 
the number of attempts. It is assumed that a discovery will fail if the service utility is 
less than 0.5. Using the CSDS with the Cloud ontology, experimental results show 
that the service utility of retrieved web-pages is well over 0.5. The results demon- 
strated that using the CSDS with the Cloud ontology, users are more successful in 
discovering Cloud services. In Fig. 7, the result show that calculating the similar- 
ity between users and a Cloud service without (numerical) reasoning for numeric 
values is hard to over 0.5 in terms of service utility in both cases. 



6 Proof-of-Concept Examples 



An example is given in this section to demonstrate functionalities of the CSDS. 

Step 0: Initially, when the CSDS is deployed, a number of Cloud providers are 
generated while each provider is posting around 25 of their services. A total of 
around 13000 web-pages existed in the virtual-www, including general web-pages. 

Step 1: The screen in Fig. 10 shows the user input query, which contains a ser- 
vice name (e.g., "Visual_Studio_2010") and requirements (e.g., OS = "Windows?," 
CPU name = "Core2Quad," CPU clock = "4.6," RAM = "9.0," HDD = "500.0," 
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Fig. 9 The best service to be founded 

network bandwidth — "5.92," network latency = "1667.0"), and mobile device sup- 
port = "No"). 

Step 2: The CSDS automatically searches with the exact service name "Vi- 
sual_Studio_2010" from the virtual-www and filters out web-pages that do not in- 
clude the "Cloud" term. The result is shown in Fig. 8. 

Step 3: The CSDS consults the Cloud ontology for service reasoning. Then, the 
similarity of each term is aggregated as the service utility. 

Step 4: The CSDS takes the highest utility, "0.8275," as the best service among 
53 web-pages and rate ordering. 

Step 5: The CSDS returns the best service (e.g., provided by "SellerVF'), which 
is the result of the service discovery as shown in Fig. 9. Additionally, results for 
the three cases, (1) without the CSDS, (2) the CSDS without the Cloud ontology, 
and (3) the CSDS with the Cloud ontology, are printed to the user interface screen. 
Other recommended services are also included in turn, as shown in Fig. 10. 



7 Conclusion and Future Work 



This paper has presented a Cloud service discovery system (CSDS). It is specially 
designed for users who want to find a Cloud service over the internet. A Cloud 
ontology is also introduced for enhancing the performance of the CSDS. The con- 
tributions of this work include (1) building of the CSDS and (2) constructing the 
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Cloud ontology. It is the first attempt in building an agent-based discovery sys- 
tem that consults an ontology when retrieving information about Cloud services. In 
present, there are few big Cloud service providers and no various services. When 
the Cloud computing will be more commonly and widely used in the near future, it 
can be helpful for Cloud users who want to find a Cloud service under their specific 
preference. 

From the empirical results in Sect. 5, the CSDS with the Cloud ontology achieved 
better performance than the CSDS without the Cloud ontology. By consulting a 
Cloud ontology to reason about the relations among Cloud services, the CSDS is 
more successful in locating Cloud services and more likely to discover Cloud ser- 
vices that meet users requirements. In addition, the CSDS essentially needs all three 
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kinds of reasoning methods to have high performance in terms of service utihty and 
success rate. Since this work is a preliminary work for Cloud service discovery, a list 
of future agendas includes the following. (1) Designing of Cloud (service) ontology 
is still an issue that is not fully addressed. (2) Although Cloud services and tradi- 
tional web service are currently described by Web Services Description Language 
(WSDL) [12], we had to design providers' services in our own model because of 
the lack of standards and experimentation. 
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Possible Applications of Navigation Tools 
in Tilings of Hyperbolic Spaces 



Maurice Margenstern 



Abstract This paper introduces a method of navigation in a large family of tilings 
of the hyperbolic plane and looks at the question of possible applications in the light 
of the few ones which were already obtained. (This paper is a revised and slightly 
extended version of a paper presented by the author at IMECS'2010, see Margen- 
stern (Proceedings of the International MultiConference of Engineers and Computer 
Scientists 2010, pp. 367-382, 2010).) 

Keywords Hyperbolic tilings ■ Cellular automata ■ Applications 



1 Introduction 

Hyperbolic geometry appeared in the first half of the 19th century, proving the in- 
dependence of the parallel axiom of Euclidean geometry. Models were devised in 
the second half of the 19th century. Here, we use one of the most popular ones, 
Poincare's disc. This model is represented by Fig. 1. 

From a famous theorem established by Poincare in the late 19th century, it is 
known that there are infinitely many tilings in the hyperbolic plane, each one gener- 
ated by the reflection of a polygon P in its sides and, recursively, in the reflection of 
the images in their sides, provided that the number p of sides of P and the number q 
of copies of P which can be put around a point A and exactly covering a neighbour- 
hood of A without overlapping satisfy the relation: + < j2' ^^^ numbers p 
and q characterize the tiling which is denoted {p,q} and the condition says that the 
considered polygons live in the hyperbolic plane. Note that the three tilings of the 
Euclidean plane which can be defined up to similarities can be characterized by the 
relation obtained by replacing < with — in the above expression. We get, in this 
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Fig. 1 Poincare's disc 
model: inside the open disc, 
the points of the hyperbolic 
plane. Lines are trace of 
diameters or circles 
orthogonal to the border of 
the disc, e.g. the line m. 
Through the point M we can 
see a line s which cuts m , two 
lines which are parallel to m : 
p and q , touching m in the 
model at P and Q 
respectively which are points 
of the border and are 
therefore called points at 
infinity. At last, and not the 
least: the line n also passes 
through M without cutting m , 
neither inside the disc nor 
outside it 





Fig. 2 Left-hand side: the pentagrid; right-hand side: the heptagrid 



way, {4, 4} for the square, {3, 6} for the equilateral triangle and {6, 3} for the regular 
hexagon. 

In the paper, we shall focus our attention on the simplest tilings which can be 
defined in this way in the hyperbolic plane: {5, 4} and {7, 3}. We call them the pen- 
tagrid and the heptagrid respectively, see Fig. 2. 

To navigate in this tilings was for a long time a non-trivial question. In 1999 
and 2000, see [4, 14], the author found a technique which allows to find one's 
way in these tilings, first in the pentagrid. A bit later, the technique was general- 
ized to the heptagrid and to infinitely many other tilings of the hyperbolic plane. 
Details and references on these results can be found in [9, 10] as well as their ex- 
tension to one tiling of the hyperbolic 3D space and another one of the hyperbolic 
4D space. 
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Fig. 3 First part of the 
splitting: around a central tile, 
fixed in advance, five sectors. 
Each of them is spanned by 
the Fibonacci tree defined in 
Fig. 4 
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Cellular automata are a tool used in various sciences, from gas statistical physics 
to economy, for simulation purposes with good results and a few industrial applica- 
tions. We refer the reader to proceedings of the last two issues of ACRI conferences 
to have a look at this range of applications. 

The navigation technique introduced in [4] allowed to implement cellular 
automata in the pentagrid and in the heptagrid and to devise a few applica- 
tions. 

In Sect. 2, we sketchily describe the navigation technique. In Sect. 3, we remind 
the results on cellular automata and in Sect. 4, we consider the applications already 
performed as well as a few others which could be useful. In Sect. 5, we conclude 
with what could be done in future work. 



2 Navigation in the Pentagrid and in the Heptagrid 



The navigation in a tiling of the hyperbolic space can be compared to the flight of 
a plane with instruments only. Indeed, we are in the same situation as a pilot in 
this image as long as the representation of the hyperbolic plane in the Euclidean 
one entails such a distortion that only a very limited part of the hyperbolic plane is 
actually visible. 

The principle of the navigation algorithms rely on two ideas which we illustrate 
on the pentagrid. The first one, is a way to split the hyperbolic plane, see Fig. 3 
and Fig. 4: the recursive structure of this splitting defines a tree which spans the 
tiling. The second idea consists in numbering the nodes of the tree level after level 
and, remarking that the number of nodes on the level n is fin+i defined by the 
Fibonacci sequence where /g = /i = 1 , to represent the numbers in the numbering 
basis defined by this sequence. Also, as the representation is not unique, we fix it by 
choosing the longest one. 
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Fig. 4 Second part of the 
splitting: splitting a sector, 
here a quarter of the 
hyperbolic plane. On the 
left-hand side: the first two 
steps of the splitting. On the 
right-hand side: making 
explicit the tree which spans a 
sector 
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Fig. 5 The Fibonacci tree: vertically, under each node, the longest representation of its number in 
the Fibonacci basis 



On Fig. 4, we can notice that the Fibonacci tree has two kinds of nodes: the white 
ones, which have three sons, and the black ones, which have two sons. In both cases, 
the leftmost son is black, the others are white. Figure 5 represents the tree in a more 
traditional way together with the numbering of the nodes and their representation 
in the Fibonacci basis. Later on, we shall call coordinate this representation of the 
number of a node. There are important properties which are illustrated by Fig. 5 and 
which 1 called the preferred son properties. It consists in the fact that for each node 
of a Fibonacci tree, among the coordinates of its sons, there is exactly one of them 
which is obtained from the coordinate of the node by appending 00. It is therefore 
called the preferred son. Moreover, the place of the preferred son is always the same 
among the sons of a node: the leftmost son for a black node, the second one for a 
white node. 
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Fig. 6 Left-hand side: first part of the splitting, around a central tile, fixed in advance, seven 
sectors. Each of them is spanned by the Fibonacci tree defined in Fig. 4, see the right-hand side 
picture. Right-hand side: the mid-point lines. This tool which shows how a sector spanned by the 
Fibonacci tree is defined in the heptagrid 



Fig. 7 The heptagrid is 
spanned by the same tree as 
the pentagrid: compare with 
the tree of Fig. 4 




From the preferred son properties, it was possible to devise an algorithm which 
computes the path from the root to a node in a linear time in the length of the 
coordinate of the node, see [5]. From this, we also get that the coordinates of the 
neighbours of a given tile can be computed from the coordinate of the tile in linear 
time too. 

It is interesting to remark that the heptagrid possesses properties which are very 
similar to those of the pentagrid, see Figs. 6 and 7. Note that the right-hand side 
picture of Fig. 6 explains why the same tree basically spans each of the seven sectors 
of the left-hand side of the figure. 

We conclude this section by mentioning that these nice properties of the penta- 
grid and of the heptagrid can be extended to two infinite families of tilings of the 
hyperbolic plane. In particular, the fact that the same tree spans the pentagrid and 
the heptagrid can be extended as follows: for each p, p > 5, the same tree spans 
the tiling {/?, 4} and the tiling {p -\-2,3]. The trees are different for different values 
of p although they share a common feature: nodes are divided into black and white 
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nodes. The difference in the number of sons from a white node to a black one is 1. 
The unique black son can be chosen to be the leftmost one. 



3 Results on Cellular Automata in Hyperbolic Spaces 

These fast algorithms allowed to implement cellular automata in the hyperbolic 
plane, first in the pentagrid and, a few years later, in the heptagrid. 

We have three kinds of results regarding cellular automata in these spaces: com- 
plexity results, universality results and a solution of two problems dealing with com- 
munications between cells of a cellular automaton. Although these results have a 
definite theoretical character, they nevertheless have a practical significance. 

The most striking result regarding complexity, is that for cellular automata in 
the hyperbolic plane, we have P/,c — NP/,c. This means that non-deterministic poly- 
nomial time computations of a cellular automaton in the hyperbolic plane can be 
performed also in polynomial time by a deterministic cellular automaton in the hy- 
perbolic plane. Moreover, the exact power of computation of the class P/,c is the 
well-known class PSPACE. The reader is referred to [2, 3, 10, 14]. 

For what is universality, there are universal cellular automata in the hyperbolic 
plane, i.e. able to simulate the computation of any Turing machine. What is more 
important is that if infinite but elementary initial configurations are allowed - within 
this limited room we cannot formally describe the exact meaning of this expression - 
it is possible to simulate any Turing machine with a cellular automaton with 9 states 
in the case of the pentagrid, see [16], and with 6 states in the case of the heptagrid, 
see [15] and even less: 4 states, see [12]. All the quoted cellular automata have a 
common structure: they simulate a railway circuit traversed by a unique locomotive 
consisting of tracks and three kinds of switches devised in [19], see also [10]. 

Now, the linear algorithm for finding the path from a node to the root of the sector 
to which the node belongs allows to devise efficient communication protocols for 
cells of a cellular automaton in the pentagrid or in the heptagrid, see [7, 8, 10]. 

If the latter point is closer to applications, the first ones also say something on this 
regard. The meaning of the complexity results is that hyperbolic cellular automata 
may run much faster than their Euclidean analogues: they have at their disposal 
an exponential area which can be constructed and used in linear time. Also, these 
computations may be universal as this is the case for their Euclidean analogues. 
And so, we can do any computations in this frame, never in more time than what 
is required for a Euclidean cellular automaton, and very often in much less time. In 
particular, in this setting, hard problems of everyday life turn out to be solvable in 
polynomial time, very often even in linear time. 



4 On the Side of Applications 

The above results might seem too beautiful to be true. However, they are theoreti- 
cal results whose proofs were checked, some of them with the help of a computer 
program, and they are correct. 
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But is this feasible? 

Our local environment is usually thought as Euclidean although it would be bet- 
ter to see it as governed by spherical geometry. Many cosmologists consider our 
universe as a space with a negative curvature. In this regard, hyperbolic geometry 
could be a better model for medium scale than Euclidean geometry. Computations 
on the orbit of Mercury are more conformal to observations when performed in a 
hyperbolic setting. So, our tools might have applications at this scale which is not a 
today urgent matter, but we know that possible tools exist. 

I would also mention another argument. An important feature of hyperbolic ge- 
ometry is the lack of similarity. As a consequence, it can be said that a shape has 
a certain size necessarily. As an example, in the hyperbolic plane, there is a unique 
size of the edge for a regular pentagon with right angles. This means that two such 
pentagons can be transformed from each other through a simple geometric transfor- 
mation which is an isometry. And so, up to isometrics, such a pentagon is unique. 
This property is shared by any figure of the hyperbolic plane. Now, if we look at 
biology, we can notice that there is no true similarity. Individual differences, for in- 
stance, are not addressed by similarity. This remark led me to [6] where I used the 
pentagrid to represent a theoretical model of living cells, the common point being 
the fact that, in both cases, a tree underlies the structures. 

Now, another field of application, more important in my eye, is provided by com- 
puter science itself. We need new concepts to handle problems raised by massive 
computations and by the management of huge amounts of information. For this, 
we need new horizons and it is not at all unreasonable to consider tilings of the 
hyperbolic plane with their navigation tools as a possible model for tackling these 
problems. Note, for instance, that trees are already used in the organisation of op- 
erating systems: many a user is faced with the tree structure of the directories of 
his/her machine. Now, trees are also spanning hyperbolic geometry and this is par- 
ticularly blatant in the field of tilings in the hyperbolic plane as I hope the reader is 
convinced after reading Sect. 2. 

In the present section, we first have a look at the existing applications, see the 
next two subsections. These applications are based on a fisher eye effect of the pen- 
tagrid and the heptagrid. Now, it is known that fisher eye techniques are of interest 
for several applications in human-machine interaction. In our last subsection, we 
again address the problem of representing, storing and exploring information which 
mainly use the coordinate system. 



4.1 The Colour Chooser 

The colour chooser is a software which was developed in my laboratory by members 
of my team, see [1, 10]. From the initial state of the chooser illustrated by Fig. 8, 
the user can navigate in the tiling in order to look after the colour he/she thinks the 
most adapted for his/her purpose. 

There are seven fixed in advance keys which indicate to which tile the user wishes 
to go from the central tile. Once the appropriate key is pressed, the chosen tile comes 
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Fig. 8 (Color online) The 
colour chooser: the pentagrid 
is also a possible tool but the 
heptagrid gives the best 
representation. Probably 
because at first glance, the 
heptagons of the figure are 
seen as hexagons: it is needed 
to count the sides in order to 
detect the difference 
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to the centre of the disc and all the other tiles move correspondingly. Consequently, 
the navigation appears as if the green disc of Fig. 8 would be a window moving 
over the hyperbolic plane. Once the black tile indicating the initial centre is no more 
visible, it is very easy to get lost. To avoid such a defect, the chooser keeps an arrow 
pointing at a point of the green disc to which the user have to go in order to go back 
to the initial centre. 



4.2 The Keyboards 

Another application was developed in the laboratory, see [18], which we consider in 
the next sub-subsection and which was developed in a different direction, as will be 
seen in the other sub-subsection of this subsection. 



4.2.1 Latin Keyboard 



The first idea was a proposal of a keyboard for cell phones. As our laboratory is in 
France, tests were performed with French students on randomly chosen sequences 
of French sentences devised for the experiment. 

Figure 9 illustrates the basic principle of the software. The idea is close to that of 
the chooser. But this time, the tiles contain letters and the user goes from the initial 
empty centre to the desired letter by pressing appropriate keys. At most three keys 
have to be pressed and with other media, pressing the keys can be replaced by three 
quick moves of the hand. 

Also, the pentagrid is used instead of the heptagrid because for this purpose it 
is more suited to the gestures of the user and letters can better be seen. The reason 
is that the regular pentagon with right angle is bigger than the rectangular hep- 
tagon with the angle ^ . This explains that letter can better be seen. This explains 
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Fig. 9 A proposal of a keyboard for cell phones 



Fig. 10 A proposal of a 
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also that less accuracy is required from the user for his/her gestures. The experi- 
ments proved that the keyboard is certainly not worse than commercial products. 
Moreover, among young people, it raised a curiosity which contributed to the quick 
learning of the keyboard. Another important feature is that we adopted a distribution 
of the letters as close as possible to the standard alphabetic order, which requires no 
additional effort of memory from the user. 



4.2.2 Japanese Keyboard 



The use of the pentagrid and the fact that hiraganas and katakanas of the Japanese 
language are traditionally presented in series based on the five vowels of the lan- 
guage inspired me the idea to use the same grid but this time, in a Japanese envi- 
ronment. We did some work in this direction, with Japanese colleagues, see [17]. 
Figure 10 illustrates the principle of the working. 

This keyboard has always met a big success in the conferences were I presented 
it in Japan. There was a prototypical implementation on actual cell phones and con- 
cluding experiments were performed with a small group of Japanese students on a 
protocol which was similar to the one followed in France but, of course, adapted to 
the Japanese language. 
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It is important to indicate that our project also aims at a full representation of the 
kanjis used in the Japanese language, starting from a phonetic approach. This is a 
very complex task and, at the present moment it is not completed. 

The question could be raised of the adaptation of the same principle to other 
languages. I was indicated by several colleagues that the occurrence of exactly 
five vowels is a common feature of many Asian languages as the languages from 
Malaysia and Philippines, also including the Polynesian languages. It would be in- 
teresting to see whether similar ideas could be developed for other languages: any 
proposal would be welcomed! 



4.3 Other Possible Applications 

Now, let us turn to other possible applications which were not yet tested and which 
are based on other principles. 

As indicated in the beginning of this section, computer science and computer 
engineering could be an important field of application of the navigation technique 
introduced by Sect. 2. 

There is not enough room to develop such applications in a detailed way. This is 
why I shall mention three possible sub-fields of application and give general argu- 
ments only in favour of these proposals. 

These three sub-fields are: the representation of the Internet, computer architec- 
ture and operating systems with data processing. 

The Internet can be represented as a graph. This is often performed in force ori- 
ented representations which assign a mass to each node and define their respective 
positions by application of the laws of mechanics. Using the pentagrid or the hep- 
tagrid can be an alternative representation. In [10], I illustrate this point by defining 
addresses of nodes which is based on the navigation technique of Sect. 2 but which 
looks like the today used IP numbers. There is another advantage in this proposal. 
For both the pentagrid and the heptagrid, [10] gives an algorithm which, from the 
coordinates of two tiles A and B, gives a shortest path from A to Z? in linear time in 
the size of the coordinates. This is an important improvement of the path algorithm 
from a node to the root of the sector of the node mentioned in Sect. 2. It also allows 
to define a fast algorithm for changing the centre of coordinates. 

This possibility may better represent the Internet connections by giving addresses 
which are continuous with respect to the connection distance between nodes. 



4.3.1 Communications Between Tiles 

In [8], I introduced a protocol of communication between tiles of the pentagrid 
or the heptagrid based on the following principle: there is an absolute system of 
coordinates but when sending a message, a tile considers itself as the central tile and 
sends a message to all tiles together with its relative address which is 0. Now each 
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Fig. 11 Illustration of the 
protocol of communication 
between tiles of the pentagrid 
or of the heptagrid. The 
circled tiles indicate a 
shortest path from Alice to 
Bob. The coordinates give 
both numbers of the edge 
shared by two tiles by giving 
the number of the side 
through which the other tile 
can be seen. The number of 
the side is defined with 
respect to that of the father in 
the tree which is 1. The other 
numbers are obtained by 
counter-clockwise turning 
around the tile 




Alice 



tile which receives the message, conveys it to its sons with respect to the coordinate 
system of the sender. It also appends an information of constant size in order to form 
the address it will convey to its sons. This allows a tile which receives the message 
and which wishes to establish a contact with the sender to send an answer to the 
appropriate sender: it simply reverses the address in a way which is described in [8]. 
Figure 1 1 illustrates the protocol on a toy example and gives the principle of the 
coordinate systems. 

There, Alice, the inhabitant of a tile sends a message to everybody. At time 8, 
Bob, who lives in another cell receives the message and decides to give an answer. 
Together with the message. Bob receives his own address in Alice's system of coor- 
dinates. Accordingly, the reversal of the address gives Alice's address in Bob's own 
system of coordinates. 



4.3.2 Internet, Processors and File Storage 

This can be transported to a set of processors: they could be organised as if they 
would stand in a disc of the pentagrid or of the heptagrid around a central one which 
would not necessarily have a control function. It is enough for that to assign them 
coordinates as those introduced in Sect. 2. The communication between processors 
could be organised according to the just described protocol. 

The shortest path algorithm can be used for the representation of the Internet but 
also for the representation of a file storage for an operating system. The system of 
coordinates which allows to construct a shortest path from one point to another in a 
linear time with respect to both coordinates should facilitate the finding of queries as 
topological neighbourhood is up to a point reflected in the coordinates themselves. 
From this remark, we can see that this can be of help also for data processing. In 
particular, constant saving mechanisms can be organised by scanning a circular area 
by a branch which moves from a central point to an indicated point of the circum- 
ference. The idea is that the branch constantly moves around this circumference and 
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that at each time, the content of the branch only is saved. After a certain time, every- 
thing is saved and updates can also be included in this constant saving provided that 
it does not exceed a fixed in advance amount. In case of exceeding the threshold, the 
update is split into as many parts as needed in order that each part should fall within 
the threshold. 

It can be noted that when the branch goes from one node of the circumference 
to the next one, in many cases, the two positions of the branch may have a long 
common interval / starting from the centre which consists of the same nodes. If no 
update occurs on the nodes belonging to / between the two corresponding tops of 
saving, then only a small part of the new branch has to be saved. This can be easily 
managed by an appropriate signalization on the branch, see [11]. 



5 Conclusion 

There is still much work to do in this domain, both in theory but also for applications. 

Practical problems are difficult as this is witnessed by the example of schedul- 
ing problems, either in airplane traffic or in production processes: such problems 
are NP-complete. We have seen that the frame proposed by this paper allows to 
solve them in polynomial time. This is not the single theoretical approach leading to 
such results. As an example, molecular computations based on a modelling of DNA 
strand reactions or on a modelling of a living cell lead to similar results. But our 
frame has the advantage of not being concerned by still unsolved biological prob- 
lems for a practical implementation, the nature of which is not yet well understood: 
either it comes from fundamental issues or it comes from not well enough mastered 
techniques. The last subsection of Sect. 4 pointed at fields where our approach might 
have feasible applications. 

It seems to me that exchanging all possible ideas is a way to find out paths which 
will turn out to give the expected solution with, in many cases, surprising outcomes. 
This paper aims at being a contribution to this large exchange. I hope that the already 
few applications explored so far will be followed by many ones. I hope that it will 
encourage people to venture along the tracks opened in this paper and, it would be 
the best, to go further towards new avenues. 



References 

1. Chelghoum, K., Margenstern, M., Martin, B., Pecci, I.: Palette hyperbolique: un outil pour 
interagir avec des ensembles de donnees (Hyperbolic chooser: a tool to interact with data sets, 
in French). In: IHM'2004, Namur (2004) 

2. Iwamoto, Ch., Margenstem, M.: Time and space complexity classes of hyperbolic cellular 
automata. lEICE Trans. Inform. Syst. 387-D(3), 700-707 (2004) 

3. Iwamoto, Ch., Margenstern, M., Morita, K., Worsch, Th.: Polynomial time cellular automata 
in the hyperbolic plane accept exactly the PSPACE languages. In: SCr2002 (2002) 

4. Margenstern, M.: New tools for cellular automata in the hyperbolic plane. I. Universal Com- 
put. Sci. 6(12), 1226-1252 (2000) 



Possible Applications of Navigation Tools in Tilings of Hyperbolic Spaces 229 

5. Margenstern, M.: Implementing cellular automata on the triangular grids of the hyperbolic 
plane for new simulation tools. In: ASTC'2003, Orlando, March 29-April 4 (2003) 

6. Margenstern, M.: Can hyperbolic geometry be of help for P systems? Lect. Notes Comput. 
Sci. 2933, 240-249 (2004) 

7. Margenstern, M.: A new way to implement cellular automata on the penta- and heptagrids. I. 
Cell. Autom. 1(1), 1-24 (2006) 

8. Margenstern, M.: On the communication between cells of a cellular automaton on the penta- 
and heptagrids of the hyperbolic plane. I. Cell. Autom. 1(3), 213-232 (2006) 

9. Margenstern, M.: Cellular Automata in Hyperbolic Spaces, vol. 1: Theory. Old City Publish- 
ing, Philadelphia (2007), 422 p 

10. Margenstern, M.: Cellular Automata in Hyperbolic Spaces, vol. 2: Theory. Old City Publish- 
ing, Philadelphia (2008), 360 p 

11. Margenstern, M.: A uniform and intrinsic proof that there are universal cellular automata in 
hyperbolic spaces. I. Cell. Autom. 3(2), 157-180 (2008) 

12. Margenstern, M.: Universal cellular automata in hyperbolic spaces. In: 13th WSEAS Interna- 
tional Conference on Computers, Rodos, July 23-25, pp. 83-89 (2009), ISBN 978-960-474- 
099-4 

13. Margenstern, M.: Navigation in tilings of the hyperbolic plane and possible applications. In: 
Lecture Notes in Engineering and Computer Science: Proceedings of the International Mul- 
tiConference of Engineers and Computer Scientists 2010, IMECS 2010, Hong Kong, 17-19 
March 2010, pp. 376-382 (2010) 

14. Margenstern, M., Morita, K.: A polynomial solution for 3-SAT in the space of cellular au- 
tomata in the hyperbolic plane. J. Universal Comput. Syst. 5, 563-573 (1999) 

15. Margenstern, M., Song, Y.: A universal cellular automaton on the ternary heptagrid. Electron. 
Notes Theoret. Comput. Sci. 223, 167-185 (2008) 

16. Margenstern, M., Song, Y.: A new universal cellular automaton on the pentagrid. Parallel 
Process. Lett. 19(2), 227-246 (2009) 

17. Margenstern, M., Martin, B., Umeo, H., Yamano, S., Nishioka, K.: A proposal for a Japanese 
keyboard on cellular phones. Lect. Notes Comput. Sci. 5191, 299-306 (2008) 

18. Martin, B.: VirHKey: a VIRtual Hyperbolic KEYboard with gesture interaction and visual 
feedback for mobile devices. In: Mobile HCI, pp. 99-106 (2005) 

19. Stewart, I.: A subway named Turing. Mathematical recreations. Sci. Am. 90-92 (1994) 



Graph Pattern Matching with Expressive 
Outerplanar Graph Patterns 



Hitoshi Yamasaki, Takashi Yamada, 
and Takayoshi Shoudai 



Abstract An outerplanar graph is a planar graph that can be embedded in the 
plane in such a way that all vertices lie on the outer boundary. Outerplanar graphs 
express many chemical compounds. An externally extensible outerplanar graph pat- 
tern (eeo-graph pattern for short) represents a graph pattern common to a finite set 
of outerplanar graphs, like a dataset of chemical compounds. The eeo-graph pat- 
tern can express a substructure common to blocks that appear in outerplanar graph 
structured data. In this paper, we propose a polynomial time algorithm for decid- 
ing whether or not a given eeo-graph pattern matches a given connected outerplanar 
graph. Moreover, we show the expressiveness of the pattern class by experiments on 
a chemical compound database. 

Keywords Graph pattern matching ■ Graph mining ■ Outerplanar graphs ■ 
Chemical graphs 



1 Introduction 

Large amounts of data that have graph structures (such as map data, CAD, biomolec- 
ular, chemical molecules, and the World Wide Web) are stored in databases. For 
example, HTML/XML documents can be expressed by ordered trees, and almost 
all chemical compounds in the NCI dataset [6], which is a popular graph mining 
dataset, are expressed by outerplanar graphs. Outerplanar graphs are planar graphs 
embedded in the plane in such a way that all vertices lie on the outer boundary. 
In Fig. 1, graphs gi, g2, gj,, G, and H are examples of outerplanar graphs. Many 
researchers are interested in acquiring knowledge from data that have structures 
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Eeo-graph pattern p 



Outeiplanar graphs 




Outerplanar graph C 



Outerplanar graph H 



Fig. 1 An eeo-graph pattern p and outerplanar graphs gi , g2: g3. G, and H over a vertex label set 
{a, b, c, d): A variable is drawn by a box with lines to its elements 



such as sequences, trees, or graphs [1, 3, 8]. Horvath et al. [4] proposed a fre- 
quent subgraph mining algorithm for outerplanar graphs. For graph mining algo- 
rithms, graph pattern matching algorithms play a key role throughout the computa- 
tions. 

For graph G, we call a maximal biconnected subgraph of G a block of G if it has 
at least three vertices. For example, in Fig. 1, Bi , . . . , fig are blocks. A block of an 
outerplanar graph has a unique planar embedding up to the mirror image. Thus the 
edges of an outerplanar graph are classified into two types; external and internal. 
External edges lie on the external face, while internal edges do not. An external 
edge that does not belong to any block is called a bridge. Because an internal edge 
must belong to a block, it is also called a diagonal of the block. For an integer d>Q, 
an outerplanar graph is said to be cf-tenuous if each of its blocks contains at most 
d diagonals. Horvath et al. [4] proposed an Apriori-like algorithm for enumerating 
all frequent cf-tenuous outerplanar subgraphs in a finite set of outerplanar graphs. In 
[7], we introduced a graph-structured pattern, called a block preserving outerplanar 
graph pattern (bpo- graph pattern for short), which is an outerplanar graph with 
structured variables, and proposed a refinement-based technique for enumerating 
all maximal frequent bpo-graph patterns in a finite set of outerplanar graphs. A bpo- 
graph pattern represents an expressive graph structure among blocks. However it 
cannot represent internal structures common to different blocks if the blocks are not 
isomorphic. 

Our final objective in this research is to propose an efficient data mining method 
for extracting more expressive outerplanar graph-structured patterns, which simul- 
taneously represent connection patterns and internal structured patterns common to 
different blocks. In this paper, we introduce a graph pattern that has structured vari- 
ables, called an externally extensible outerplanar graph pattern {eeo-graph pattern 
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for short). A variable of an eeo-graph pattern is a vertex pair that is not currently 
an edge in the graph and that becomes an external edge if added as an edge to the 
graph. Then a variable of an eeo-graph pattern is called an external variable. Ex- 
ternal variables can be replaced with any connected outerplanar graphs that satisfy 
certain conditions. In Fig. 1, a graph pattern p is an example of eeo-graph patterns, 
and an outerplanar graph G is obtained from p by removing variables h\, h2, and 
/z3, and identifying vertex pairs (mi, uj), (M3, M4), and (M5, Mg) of p with (v[, V2) 
of gi, (wi, W2) of g2, and (zi, zi) of ^3, respectively. For an eeo-graph pattern p 
and a connected outerplanar graph G, we say that p matches G if there is such a 
variable replacement by which the graph obtained from p is subgraph isomorphic 
to G under the condition of preserving blocks. In this paper, we propose a poly- 
nomial time algorithm for the pattern matching problem for the class of eeo-graph 
patterns. Moreover, we show the expressiveness of the pattern class by experiments 
on a chemical compound database. 



2 Externally Extensible Outerplanar Graph Patterns 

Let G = (V, E) be a graph. Let A, A, and X be infinite alphabets, where (A U 
A) n X = 0. A variable of G is a list of different vertices of V , which is denoted by 
(di, . . . , vi) (t > 1), where vt eV (l <i <i) and Vj ^ Vj if / / 7 (1 < /, ; < £). 
All vertices and edges are labelled with symbols in A and A, respectively. In this 
paper, all variables are assumed to be labelled with mutually distinct symbols in X. 
Then a triple p — (V , E , H) is called a graph pattern if (V, E) is a graph and H is 
a set of variables of {V, E). For a set or a list S,\S\ denotes the number of elements 
in S. Let p be a graph pattern. Also, V{p), E{p), and H{p) denote the sets of all 
vertices, edges, and variables of p, respectively. Moreover, Xp(u), Sp(e), and Xp(h) 
denote the vertex label of m e V(p), the edge label of e e E(p), and the variable 
label of h e H(p), respectively. 

A graph pattern p' is said to be a subgraph pattern of p if V(p') c V{p), 
E(p') C E(p), and H(p') C H{p). Below we regard a standard graph as a graph 
pattern with no variable. Let p and q be graph patterns. We say that p is isomor- 
phic to q if there is a bijection i/f : V{p) -^ V(q) such that (1) for any v e V(p), 
Ap(i;) = Ag(Vf(D)), (2) {m, i>} e £(p) if andonly if {iA(m), iA(i;)} e £(g), (3)forany 
{m, v] e E{p), Sp({u, v}) = Sq({f(u), fiv)}), and (4) for I > I, ivuV2, ■ ■ ■ , vt) e 
H(p) if and only if (-[//(vi), ir(v2), ■ ■ ■ , i^ivi)) e H(q). We say that p is subgraph 
isomorphic to q if p is isomorphic to a subgraph pattern of q . 

In an outerplanar embedding of an outerplanar graph G, an edge of G is external 
if it borders the outer face. 

Definition 1 A graph pattern p is said to be an externally extensible outerplanar 
graph pattern {eeo-graph pattern for short) if p satisfies the following conditions: 
(1) Every variable has exactly 2 vertices, (2) E{p) n Enip) = 0, where Enip) — 
{{m, v] I (u, v) e H(p)}, and (3) the graph Gp = (V(p), E{p) U Eh(p)) of p is a 
connected outerplanar graph, and all edges in Eh (p) are external. 
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Graph pattern p Graph G 

Fig. 2 A graph pattern p and graphs g[, gi, G over a vertex label set {a, b, c, d) 

The variables of an eeo-graph pattern p are called external variables. Moreover, 
an external variable h — (u, v) in p is called a bridge variable if {u, v] is a bridge in 
Gp, and a block variable otherwise. If Gp is biconnected, p is called a biconnected 
eeo-graph pattern. 



Definition 2 Let x be a variable label in X. Let p and q be eeo-graph patterns and 
h = (vi,V2) a variable of p labelled with x . Leta = («i, M2) be a list of two distinct 
vertices in q. The form x := [^, cr] is called an eeo-binding for x if (1) Xq{ui) — 
Ap(iii) and Xqiuj) — ^p(v2), and (2) if /z is a block variable of p, all edges on a 
path between mi and 1/2 are bridges. 

A new graph pattern p{x :— [q,cr]} is obtained by applying the eeo-binding 
X :— [q, a] to p in the following way. Let h — {vi, vj) be a variable in p with the 
variable label x. For /z = (iij, 112), we attach q to p by removing h from H(p) and 
by identifying 11 1 and V2 with mi and U2 of q, respectively. The new graph pattern 
p{x :—[q,a]} is also an eeo-graph pattern. 

Let p,qi, . . . ,q,„ be eeo-graph patterns. An eeo-substitution for /? is a finite 
collection of eeo-bindings {xi := [gi, cti], . . . , x^ := [^m> fm]} where xi, . . . , x,„ 
are mutually distinct variable labels in X and each gi (1 < / < m) has no variable 
labelled with a variable label in {xi, . . . , x„,}. For an eeo-graph pattern p and an 
eeo-substitution 9 for p, p6 denotes the eeo-graph pattern obtained from p and 6 
by applying all the eeo-bindings in to p simultaneously. 

We give an example of eeo-substitutions. In Fig. 2, we give a graph pattern p 
that has variables hi = (mi, M2) and h2 — (M3, M4), so that the graph p{xp(hi) :— 
[gi, (vi,V2)],Xpih2) := [g2, (wi, W2)]} is isomorphic to G. 

Given a graph pattern p and a graph G, an operator that decides whether or 
not p matches G is called a matching operator. An isomorphism (resp. a subgraph 
isomorphism) matching operator returns a "true" if there is a substitution 6 such 
that p9 is isomorphic (resp. subgraph isomorphic) to G, and "false" otherwise. Be- 
cause the subgraph isomorphism problem for outerplanar graphs is NP-complete, 
the problem of deciding whether a subgraph isomorphism matching operator returns 
a "true" or "false" is hard to solve. On the other hand, the subgraph isomorphism 
problem for biconnected outerplanar graphs is solvable in cubic time [5]. In this 
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paper, we use a restricted subgraph isomorphism introduced by [4] as a matching 
operator. 

Definition 3 For two graphs G and H, we say that G is bridge and block preserving 
(BBP) subgraph isomorphic to H if there is a subgraph isomorphism \jf from G to 
H that maps (i) the set of bridges of G to the set of bridges of H and (ii) different 
blocks of G to different blocks of H. 

For an eeo-graph pattern p and an outerplanar graph G, we say that p matches 
G if there is an eeo-substitution 6 such that p6 is BBP subgraph isomorphic to G. 

For example, in Fig. 1, from p and 6 — {xp{hi) :— {g\, {v\, 1)2)], Xpihj) '■— 
[g2, iwi, wj)}, Xp{hj,) := [g3, (zi, 22)]}, we obtain the outerplanar graph p6 that is 
isomorphic to G. The graph G is BBP subgraph isomorphic to H such that blocks 
B\, B2, B3, and B4 of G correspond to Bs, Be, Bq, and Bg of //, respectively. 
Therefore, p matches both G and H. 



3 A Matching Algorithm for EEO-Graph Patterns 

First of all, we give an outline of a polynomial time algorithm for solving the match- 
ing problem for a given eeo-graph pattern p and a given outerplanar graph G. The 
idea of our algorithm for the matching problem is similar to the polynomial time 
matching algorithm for block preserving graph patterns in the works of Sasaki et 
al. [7] and Yamasaki et al. [10]. First we transform p and G into labelled tree struc- 
ture representations t{p) and T{G), respectively, and fix one vertex of t{p) as its 
root. Next, for every vertex of T(G), we specify it as the root of T{G), and proceed 
to construct correspondences between all vertices T{G) and t(p) in the bottom up 
manner, that is, from the leaves to the root of T(G). In the process of the algorithm, 
we have to decide whether or not a block of G is matched by a subgraph pattern of 
p that has block variables. 

An eeo-graph pattern p with a distinguished vertex r e V{p) is called a rooted 
eeo-graph pattern and denoted by p'^ . Similarly, an outerplanar graph G with a dis- 
tinguished vertex s e V{G) is called a rooted outerplanar graph and denoted by 
G ' . We say that a rooted eeo-graph pattern p'' matches a rooted outerplanar graph 
G'^ if p matches G, so that r is mapped to s. EOVa.a (resp. Oa,a) denotes the 
set of all eeo-graph patterns (resp. the set of all connected outerplanar graphs) over 
a vertex label set A and an edge label set A. Moreover, SOV^ ^ (resp. O'^ ^) 
denotes the set of all rooted biconnected eeo-graph patterns (resp. the set of all 
rooted biconnected outerplanar graphs) over a vertex label set A and an edge label 
set A. 

We give a polynomial time algorithm for solving the following problem. 
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Matching Problem for Rooted Biconnected EEO-Graph Patterns 
Input: A rooted biconnected eeo-graph pattern p'' e EOVJ ^ and a 

nected outerplanar graph G* e ^aa- 
Problem: Decide whether or not p'' matches G*. 



For p' e EOVjl^ ^ and G-' e 0^% , let « = | y (p'' ) I and A? = | y (G' ) | . Hereafter, 
we assume that y(p'') = {1,2,... , «} with r = 1 and V{G') = {1, 2, . . . , A?} with 
s = 1, and that sequences (1, 2, . . . , «) and (1, 2, . . . , A^) are clockwise or counter- 
clockwise orders of boundaries of outerplanar embeddings of p'' and G ' . 

Lemma 4 For p'' e EOV^^ ^ and G^ e O''^ ^, if p'' matches G\ there is an eeo- 
substitution 6 for p'' , and a subgraph isomorphism i/f : V(p0) — >■ V(G) such that 
xl/{l) — 1 and the vertex sequence i^{2), ir(3), . . . , xl/(n) is either an increasing or 
decreasing number sequence. 

Proof Let C = (/i , (2, ■ ■ ■ , 'm) be a simple cycle of G* such that ;i = 1. We assume 
that there is an index k such that 2 < k < m, ij, ■ ■ ■ ,ik is an increasing number 
sequence, and ik+\ < '*:■ Then, there is an index h such that \ <h < k and /;, < 
ik+i < ih+i- On the outer cycle of G^, there is a path reaching from if, to 4+1 > 
denoted by P[ , and a path reaching from i^_|_i to ih+\, denoted by P2. Let P3 be the 
path (ik+i, . . . , /„,, /i) on C, and P4 the path (iic,ik + l, . . . , N,l) on the outer cycle 
of G' . Since the outer cycle of G* and C are simple cycles, P3 does not contain i^ 
and P4 does not contain ik+i- Since P3 and P4 share at least one vertex, let £ be the 
first cross point of Pj and P4. Let P5 be a path consisting of £,..., A^ on the outer 
cycle of G* and f'l, . . . , ih+i on C. 

Suppose that k ^ h + 1. We note that P5 does not contain i^ and ik+i- Let P^ be 
the path (//,+i , ih+2, ■ ■ • . 4) on C. Then a subgraph of G', which consists of P2, P3, 
P4, P5, Pe and the edge [i^, ik+i), is homeomorphic to K4 (the complete graph on 
four vertices). No outerplanar graph contains any subgraph homeomorphic to K4. 
Suppose that k = h + 1. We note that //, > ii and //, 7^ £. Then, a subgraph of G' 
consisting of Pi, P3, P4, P5, and the edge {ik, ik+i] is homeomorphic to K4. This 
contradicts the fact that G^ is an outerplanar graph. Therefore, i2, ■ ■ ■ , im is either 
an increasing or decreasing number sequence. 

From this fact, for any biconnected outerplanar subgraph g of G' containing s, 
the outer cycle of g has the vertex sequence as an increasing number sequence. From 
the definition of eeo-graph patterns, since the order of vertices of the outer cycle of 
p'^ is not changed by any eeo-substitution, consequently, the lemma holds. D 

For vertices ( and /' (( < /') in p'' , p'^[i, i'] denotes the subgraph pattern obtained 
from the induced subgraph p[{i, / + 1, . . . , ;'}] by removing a variable (/, i') e H(p) 
if it exists. For example, in Fig. 3 we give a rooted biconnected eeo-graph pattern p'' 
and its subgraph patterns p'^{\, 5] and p'^{5, 8]. For vertices / and (' (/ < /') in /?'', 
and vertices j and j' (j < j')m G*, we say that/?'' [i, /'] matches G^[j, j'] if there is 
an eeo-substitution 9 for p''[i, i'] and a subgraph isomorphism x// : V(p'^[i, i']6) -^ 
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M 

\J/ l^'-' 

6 

Fig. 3 A rooted biconnected eeo-graph pattern and its subgraph patterns 



Y(G^\j, _/']) such that Vf(/) = j and i/^(/') — j' . For vertices / and ;' (/ < i') in /?'', 
the correspondence-set (C-set for short) of the pair (/, i'), denoted by CS{i, i'), is 
the set of all pairs (j, j') of vertices in G' such that p''[i, i'] matches G'[y', j']. 

Lemma 5 For a vertex i in p' where 1 < / < n, and vertices j and j' (j < j') in 
G^'^ (j' j') ^ CS(i, i + 1) if and only if either of the following conditions holds. 

1. {i, / + 1} e £{/), {;, /} e £(G'), Xp{i) = kdj), Xpd + 1) = Xg(J'), and 

Sp({i,i + l})=SG({jj'}). 

2. (i, / + 1) e H(y), Xp(i) = XgU), and kp{i + 1) = X{j'). 

Proof If {/, i + 1} e E{p''), it is easy to see that statement (1) holds. If (/, / + 1) e 
H{p''), p''[i, i + 1] consists of two vertices, / and / + 1, and has no edge and no 
variable. Therefore, statement (2) holds. D 

Statement (2) of the above lemma means a variable (/, / + 1) supplements any 
subgraph G'^[j, j'] such that Xp(i) = ^g(J) and kp(i + 1) = k{j'). We note that 
any variable in p'' forms either (/, i + 1) (1 < / < n) or (1, «). 

Let / and /' be vertices in p'' such that ;' — / > 1 and they border on the same 
inner face; in other words, if an edge {/, i'} exists, {/, i'} maintains the outerplanarity 
of p'' . The chordless path of p''[i,i'], denoted by P(p''[i,i']), is a simple path 
(i'l , (2, ■ ■ ■ , im) such that iiJj, ■ • • > hn is an increasing number sequence satisfying 
that m > 2, ii — i, i„, — i', and, except for edges and variables constituting the path, 
there is possibly only one edge {ii, i,„} or variable (ii, i,„) between all vertices in 
the path. 

For example, in Fig. 3, we can see that chordless paths of p'^[l, 5] and p'^[5, 8] 
are (1, 2, 3, 5) and (5, 6, 8), respectively. It is easy to show the following lemma. 

Lemma 6 Let i and i' be vertices in p'^ such that i' — i > 1 and they border on a 
same inner face, and let P{p'^[i, i']) — (i'l, i2, . . . , im)- If {'\ i'} ^ E{p''), then the 
vertices ii,i2, ■ ■ ■ , im i^re cutpoints of p'^[i, i']. 
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Lemma 7 Let i and i' be vertices in p'^ such that i' — i > 1 and they border on a 
same inner face, and j and j' vertices in G* such that j < j' . Let P(p''[i, /']) — 
(il ,i2,---,im)- Then, {j, j') e CS(i, i') if and only if the following conditions hold. 

1. //{/, /'} e Eip'-), {j, j'} e E(G-^) and Sp({i, i'}) = Sg({j, j'}). 

2. There is a vertex j" in G" such that j < j" < j', (j, j") e CS{ii, im-l) '^nd 

(j",j')eCS(i,n-l,im). 

Proof Suppose that the if-statement holds. There is an eeo- substitution di for 
p'^[ii, im-i] (resp. 02 for p''{i„-\, /,„]), and a subgraph isomorphism i/^i from g\ 
to G'{j,i"] where gi = p''[ii,i,n-\\e\ (resp. 1/^2 from g2 to G'[j",j'] where 
g2 = p''[im-i,im]02) such that VfiO'i) = ;■ and iri(im-[) = ;'" (resp. \l/2{im-\) = /' 
and if2{im) — j')- Let 6 be the union of 9i and 62, and g = p''[i, i']6. Let \[r be the 
injection from V{g) to V(G^[j, j']) such that ir(k) = 1/^1 (^) for each k e V(gi) 
and i/(k) = 4r2{k) for each Ic e V(g2)- Because statement (1) holds and, from 
Lemma 6, there is possibly only one edge (/,/') between V{gi) — {(„,-!} and 
V(g2) — {im-i}, we see that for all vertices k and k' in g, if {A:, k'} e E{g), then 
{f(k), fik')} e E(G') where Sg([k,k'}) = Sailfik), ^(k')}). Therefore, 1/^ is a 
subgraph isomorphism from g to G'[y, /]. 

Conversely, suppose that there is an eeo-substitution 6 for p'^[i, /'], and a sub- 
graph isomorphism i/r from g to G^[j, j'] where g — p''[i, i']6, such that i/f (/) — j 
and ir{i') — j'. It is easy to see that statement (1) holds. By assuming {;,;'} G 
E(p''), we regard g and G'[y, j'] as biconnected outerplanar graphs. We suppose 
that all vertices on the outer cycle of g are identified as 1,2, ... ,i such that / and 
i' are identified as 1 and i, respectively, and i,n-i is identified as I' in g. From 
Lemma 4, the vertex sequence (ilr (I) , -[[r (2) , . . . , -tp- (£)) is an increasing number 
sequence. Let j" = ir{i'), and g[l,^'] and g[l',l] will be subgraph isomorphic 
to G''[_/,y"] and G*[y",y'], respectively. Let 61 and 62 be subsets of 9, which 
consist of all eeo-bindings for variables in p'^[ii, im-i] and p''[im-\, im], respec- 
tively. Then, g[l,€'] (resp. g[l',i]) is obtained from p''[ii,i„,-i] and di (resp. 
p'lim-iJm] and 62). Accordingly, p''[iy,i,n-\] matches G'ij, j"] and p'lim-iJm] 
matches G" [/',/]. D 

Lemma 8 Let {i, i'} e E(p'') be a diagonal in p'' such that i < i' , and j and j' 
vertices in G" such that j < j' . Let P(p'^[i, /']) = {ii, ('2, ■ • ■ , im)- Then, {j, j') e 
CS(i,i') if and only if lj,j'} € -£'(G*) where Sp(li,i'}) — SgHJ^J']) '^'"^ there 
is an increasing number sequence {j\, J2, ... , jm) such that j\ — j, j,„ — j', and 
(jk, jk+l) e CS{ik, ik+\)for each k (l<k <m). 

Proof It is immediate from Lemma 7. D 

Lemma 9 For p'' e EOVJl ^ and G" e O''^ ^, p'' matches G* and the vertex 
sequence (1,2, ... ,n) is mapped to an increasing number sequence by an sub- 
graph isomorphism \jr : V(p'^9) -^ V{G''^) for an eeo-substitution 9 if and only if 
(1, i) e CS{\,n) for a vertex i e V(G'). 



Graph Pattern Matching with Expressive Outerplanar Graph Patterns 239 

Algorithm: MatchBlock-^ OP'^^^; 
Input: p'' 6 SOV^ ^ and G' eO'j^; 
Output: TRUE or FALSE; 
begin 

1: Let« = |V(p'')|andA^ = |V(G-')|; 

2: Let (1, . . . , n) (resp. (1, . . . , A')) be the clockwise order of boundary of p'' (resp. G*); 

We assume that r = 1 and i = L 
3: foreach (i, i') with I <i <i' <n s.t. {;, ;'} e E{p'') or (;, ;') e Hip'') do begin 
4: CSd{i,i')--<A\ 
5: if i' = / + 1 and {;, /') e E{p'') then 
6: foreach {;, /} e E{G') s.t. j < /, i <j <i + N - n, i' < / <j' + N-n, 

>~p(i) = ^gO'). ^pO") = ^gO") and 5p(ii, /'}) = SciU, j']) do 
7: addO-,/)toC5rfa,/') 

8: else If ;' = i + l and (/, /') e H(p'') then 
9: foreach (j, j') with y < j', i S j Si + N — n, 

i' S j' Sj' + N-n, kpii) = AgO'), and kp{i') = ^g(/) do 
10: add (y,/) to C5rf (/,/') 

11: end; 

12: Let S is an empty stack; 

13: for /':=«- 1 downto 3 do If {1, /'} e £(p'') then push((l, ;'), S); 
14: for ! := 2 to n - 1 do 

15: for i' := n downto ; + 2 do If {/, /'} e E(p'') then push((;, ;'), S); 
16: while S is not empty do begin 
17: (j, (•') := pop(S); 
18: foreach {;, /} eE(p'') s.t. / < j < « + A^ - n and 

/ = max{yt \j<k, i'<k<i' + N-n and {j, k} e E(G^)} do begm 
19: C ■- MATCHSUBBLOCKCpqi, i'], G'[j, /]); 

20: foreach k e C do add (;, k) to CSdii, i') 
21: end 
22: end; 

23: C := MATCHSUBBLOCK(/[l,n], G"[l, W]); 
24: if C 7^ then return TRUE; 

25 : Let ( 1 , . . . , n) be the counterclockwise order of boundary of p^ ; 
26: Do lines 3-24; 
27: return FALSE 
end. 

Fig. 4 A pattern matching algorithm for biconnected eeo-graph patterns 

Proof If {1,m} e E(p''), it is easy to see that the lemma holds. If (1, «) e H(p'^), the 
variable (1, n) can supplement the path {i,i + \, . . . ,N , 1), therefore, p'' matches 
G'. D 

Using Lemmas 4-9, we give an algorithm for solving the matching problem for 
biconnected eeo-graph patterns. For any vertex i in p'' , any vertex j in G' cor- 
responding to / satisfies i < j < i + N — n. Therefore, to decide whether or not 
/?'' matches G^, we only need to compute subsets of C-set, CSd(i, i') = {(j, j') e 
CS{i, i') \i <j <i + N -n and i' < j' <i' + N -n] for each {/, ;'} e Eip') 
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Procedure: MatchSubblock; 

Input: an eeo-subgraph pattern p'^[i, i'] and an outerplanar subgraph G^[j, j']; 

Output: the set [k e ViG") \j<k<j' and //[;, ('] matches G'[j, k]}; 

begin 

1: 
2: 
3: 
4: 
5 
6 
7: 



P{p''[i, i ]) := (i\,i2, ■■■, im)\ II iy = ', 'm = ' ■ 

for £ := 2 to m do begin 

CSijiii) ■- 0; // CSijiii) means the set [k e V(G') | (;, k) e CSdiiiJi)}- 
foreach k e CS, j(ii-i) do 
foreach (k, k') e CSaiii-i, ii) do 
if £ < m then add k' to CSij (ii) 
elseif {;i,/m) e Eip') and [j,k'} e E{G') s.t. 5p({ii, /,«}) = %({j,*'}) then 

add A:' to CSij(im) 
else if (I'l , i,n) e H(p'') then add /:' to C5,_ y {im) 
end; 
return C5,- ,(;■') 



9 

10 

11 

12 

end. 

Fig. 5 A procedure for computing a subset of CSdii, i') with respect to Clj, j'] 



and (/, (') e H{p''). Hence, we give a pattern matching algorithm MatchBlock- 
gQjyb ^ in Fig. 4 for computing the sets CSd(i, i') for all edges {/, i'} e E(p'') 
and variables (;', /') e H(p'') by using a dynamic programming manner The algo- 
rithm assigns all C-sets of pair (i, i + 1) where 1 < / < n first, and it assigns each 
C-set of pair (i, i') such that {(, /'} e E{p^) is a diagonal and for its chordless path 
{i\ ,i2, ■ ■ ■ , im), the C-sets of all pairs {ik, ik+\) have been already assigned for all 
k {\ <k < m). For an eeo-subgraph pattern p'^{i, ('] and an outerplanar subgraph 
G'[y, /], Procedure MatchSubblock in Fig. 5 computes a subset of CSd{i, i'), 
the set {(j, k) e CSd(i, i') \ k < j'}. The assignment of C-sets terminates when the 
C-set of pair (l,n) is assigned, and if (1, ^ CSd(l, n) for all ; e V(g'), we iden- 
tify vertices of p'' in reverse and assign C-sets again. 
Then, we have the following lemma. 

Lemma 10 For a rooted biconnected eeo-graph pattern p'' and a rooted bicon- 
nected outerplanar graph G^ , the problem of deciding whether or not p'' matches 
G^ can be correctly solved in 0{nN^) time, where n — \V(p)\ and N — \ViG)\. 

Proof Since \E(p)\ + \Hip)\ < 2\V(p)\ - 3, we need 0{n(N - nf) time at the 
initial stage (lines 3-11) of MatchBlock-£07"^''^. Let p''[i, i'] be a subgraph 
pattern of p'' and P(p''[i, i']) — (iiJi, ■■■, im)- In Procedure MATCHSUBBLOCK, 
each vertex ii (I < I < m) is assigned at most N — n vertices of G% and therefore, 
MatchSubblock works in 0(m{N - «)) time. Since the total lengths of chord- 
less paths of diagonals in p'' is \E{p)\ + \H{p)\ — 1, we need 0(nN{N — «)) time 
at lines 16-22 of MatchBlock-^ CP^*^. Therefore, the total time is O(nN^) 
time. D 
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Theorem 11 The matching problem for an eeo-graph pattern p e £OV a,a and an 
outerplanar graph G e Oa,A is computable in 0{nN ) time, where n and N are 
the numbers of vertices of p and G, respectively. 

Proof Let u and v be vertices of rooted eeo-graph pattern p'' and rooted outerplanar 
graph G' [v], respectively. From Lemma 10 and the proof of Corollary 1 in the work 
of Yamasaki et al. [10], to decide whether or not p'^[u] matches G''[v], we need 
O(cuC^) time where Cu and Cy are the numbers of children of u and v, respectively. 
Then, the runtime for deciding whether or not p'' matches G" is O(nN^) time, and 
consequently, the total time is 0(nN^) time. D 



4 Frequent Pattern Mining on Chemical Datasets 

To show the expressiveness of our graph pattern class £0'Pa,a^ we have im- 
plemented our matching algorithm with a frequent pattern mining method of Ya- 
masaki et al. [10], and experimented on subsets of the DTP AIDS antiviral screen 
database [2], which contains 42,689 chemical compounds. 

Let 5 be a finite subset of Oa,a and p a eeo-graph pattern in SOPa.a- 
Then, Osip) denotes the set of outerplanar graphs in S that are matched by 
p. The frequency of p with respect to S, denoted by suppg(p), is defined as 
suppg(p) — \Os{p)\/\S\. Let f be a real number where < f < 1. A eeo-graph 
pattern p e SOV a.a is t -frequent with respect to S if supp^(p) > t. We call this 
real number t a frequency threshold. 

The number of chemical compounds that can be expressed by connected outer- 
planar graphs is 35,515 (about 83.2%). The DTP AIDS Antiviral Screen program 
has checked tens of thousands of chemical compounds for evidence of anti-HIV 
activity. Compounds in the database are classified into three categories, confirmed 
active (CA), confirmed moderately active (CM), and confirmed inactive (CI). We 
used a set of 302 CA compounds, denoted by ^ca, all of which can be expressed 
by connected outerplanar graphs. We applied our method to the set iS'ca with fre- 
quency threshold 0.3, and generated frequent bpo-graph patterns [7] and eeo-graph 
patterns. From discovered bpo-graph patterns (resp. eeo-graph patterns), we ex- 
tracted graph patterns that are maximal with respect to frequency threshold t, i.e., 
f-frequent bpo-graph patterns (resp. eeo-graph patterns) p satisfying that there is 
no f-frequent bpo-graph pattern (resp. eeo-graph patterns) q such that L{q) C L{p) 
and suppg^^(p) — suppg^^(q), where L(p) — [G e Oa.a I P matches G}. 

We found 836 frequent bpo-graph patterns and 18,620 frequent eeo-graph pat- 
terns that are maximal with respect to frequencies. Moreover, from frequent eeo- 
graph patterns, we picked out eeo-graph patterns with no variable, which corre- 
spond to subgraphs obtained by applying a frequent outerplanar subgraph mining 
algorithm of Horvath et al. [4]. There are 28 of these subgraphs. For these three 
types of graph patterns, we computed frequencies with respect to a set of 34,369 CI 
compounds, denoted by Sci, all of which can be expressed by connected outerpla- 
nar graphs. We extracted graph patterns the frequencies of which for Sci are lower 
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Fig. 6 The top 10 scores for each type of graph pattern. For a graph pattern p, the score of p is 





score 0.472 



score 0.424 




score 0.303 
Fig. 7 Examples of eeo-graph patterns discovered in a dataset of the DTP AIDS antiviral screen 



than 0.05. Accordingly, we obtained 15 bpo-graph patterns, 4,113 eeo-graph pat- 
terns, and 7 subgraphs. The maximum numbers of vertices of obtained bpo-graph 
patterns, eeo-graph patterns, and subgraphs are 9, 15, and 8, respectively. 

For each obtained graph pattern p, we computed suppg^^ (p)—supp^^^ (p), called 
the score of p. Figure 6 shows the top 10 scores for each type of graph pattern. The 
number of obtained bpo-graph patterns is larger than the number of subgraphs, but 
there is little difference between top scores of these two types of graph patterns. The 
number of obtained eeo-graph patterns is larger than the numbers of the other two 
types of graph patterns, and top scores of eeo-graph patterns are higher. The result 
shows that eeo-graph patterns can capture more detailed characteristics common to 
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active data than other outerplanar graph patterns. We show examples of discovered 
eeo-graph patterns in Fig. 7. 



5 Conclusions 

In this paper, we introduced externally extensible outerplanar graph patterns (eeo- 
graph patterns) and gave a polynomial time algorithm for deciding whether or not 
a given eeo-graph pattern matches a given connected outerplanar graph. Moreover, 
we have evaluated the expressiveness of the graph pattern class by experiments on 
chemical compound database. For future works aiming at efficient graph mining sys- 
tems for real-world databases, we are studying graph pattern representations based 
on graph transformation systems. 
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Setvectors - An Efficient Method to Predict 
Cache Contention 



Michael Zwick 



Abstract In this chapter, I present a new method called Setvectors to predict cache 
contention introduced by co-scheduled applications on a multicore processor sys- 
tem. Additionally, I propose a new metric to compare cache contention prediction 
methods. Applying this metric, I demonstrate that the Setvector method predicts 
cache contention with about the same accuracy as the most accurate state-of-the-art 
method. However, the Setvector method executes nearly 4000 times as fast. This 
chapter is a revised and extended version of Zwick et al. (Proceedings of the Inter- 
national MultiConference of Engineers and Computer Scientists 2010, pp. 244-251, 
2010), presented at the MultiConference of Engineers and Computer Scientists in 
Hong Kong. 

Keywords Cache contention ■ Co-scheduling ■ Setvectors 



1 Introduction 

With multicore processors, chip manufacturers try to satisfy the ever increasing de- 
mand for computational power by parallelization on thread or process basis, making 
performance of computer systems more and more independent from the saturated 
processor clock speed. However, one important limitation that does not rely on pro- 
cessor clock speed, but on the computational power of the processor, is the ever 
increasing processor memory gap: Although both, processor and DRAM perfor- 
mance, grow exponentially over time, the performance difference between proces- 
sor and DRAM grows exponentially, too. This happens due to the fact that "the 
exponent for processors is substantially larger than that for DRAMs" [7] and "the 
difference between diverging exponentials also grows exponentially" [7]. 
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A way to deal with the exponentially diverging memory gap is to transform com- 
putational performance into memory hierarchy performance, making memory per- 
formance not only benefit from improvements of the memory hierarchy system, but 
also from better (and in a much higher rate evolving) processor technology. One 
possibility therefore is to spend computational power to find good application co- 
schedules that minimize overall cache contention. Reducing DRAM accesses by 
optimizing cache performance is a key issue in today's and tomorrow's computer 
architectures. 

Fedorova [2] identified L2 cache performance as a most crucial factor regarding 
overall performance degradation in multicore processors. Figure 1 shows the effect 
of L2 cache contention on the SPEC2006 benchmark mile, running on one core 
of a dual core processor, while each of the applications astar, gcc, bzip2, gobmk 
and Ibm is executed on the other core. It can easily be seen that the performance 
of mile heavily degrades when co-scheduled with the Ibm benchmark, while other 
co-schedules have a much lower performance burden. 

The most important requirement in order to optimize co-schedules for cache con- 
tention is a good method to predict cache contention of application co-schedules 
from specific application characteristics. Although a number of techniques have 
been investigated to predict L2 cache performance from some application char- 
acteristics for single core processors, only little effort has been spent so far to 
predict L2 cache performance of co-scheduled applications in a multicore sce- 
nario. 

In this chapter, I propose a new method called Setveetors to predict cache con- 
tention in multicore processors. I compare this method to the Activity vectors pro- 
posed by Settle et al. [6] and the circular sequence based Prob model presented by 
Chandra et al. [1]. I show that the Setvector approach predicts optimal co-schedules 
with about the same accuracy as the best performing circular sequence based method 
does; however, it executes about 4000 times as fast. The remainder of this chapter is 
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organized as follows: Section 2 presents state-of-the-art techniques to predict cache 
contention; Section 3 introduces the Setvector method. In Sect. 4, 1 propose a new 
metric called MRD (mean ranking difference) to compare cache contention predic- 
tion techniques and discuss the parameters applied in the simulation. In Sect. 5, I 
present the results. Section 6 concludes this chapter. 



2 State-of-the-Art Techniques to Predict Cache Contention 

In this section, I present state-of-the-art techniques to predict cache contention in 
multiprocessor systems, namely Activity vectors introduced by Settle et al. [6] and 
Dhruba Chandra et al.'s stack distance based FOA (frequency of access) and SDC 
(stack distance competition) model [1] and the circular sequence based Prob (prob- 
ability) model [1]. 



2.1 Settle et al.'s Activity Vectors 

Alex Settle et al. studied processor cache activity and observed that "program be- 
havior changes not only temporally, but also spatially with some regions hosting the 
majority of the overall cache activity" [6]. To exploit spatial behavior of cache ac- 
tivity to estimate cache contention, they divide the cache address space into groups 
of 32 so-called super-sets and count the number of accesses to each such super set. 
If, in a given time interval, the accesses to a super set exceed a predefined threshold, 
a corresponding bit in the so-called Activity vector is set to mark that super set as 
active. 

To predict the optimal co-schedule B, C or D for a thread A, every bit in the 
Activity vector of A is logically AND-ed with the corresponding bit in each appli- 
cation B, C and D. The bits resulting from that operation are summed up for each 
thread combination A ^r^ B, A ^^ C and A -f> D. As a co-schedule for A, Settle et 
al. select that thread in {B, C, D} that yields the lowest value. Besides access count 
information. Settle et al. additionally apply miss information in the Activity vectors. 
To further distinguish Settle's method from the circular sequence method however, 
I only regarded cache access information for my simulations; the presented results 
might look different, if miss rate information would be included in the Activity vec- 
tors. 



2.2 Chandra et al.'s Stack Distance Based FOA and SDC Methods 

Chandra et al. [1] propose to use stack distances to predict cache contention of 
co-scheduled tasks. Stack distances have originally been introduced by Mattson 
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Fig. 2 Stack distance 
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et al. [5] to assist in the design of efficient storage hierarchies in virtual memory 
systems. Hill and Smith [3] showed that they can also be easily applied to evaluate 
cache memory systems. 

The method assumes a cache with LRU (least recently used) replacement policy 
and works as follows: Given a cache with associativity a, the number of a + 1 coun- 
ters C\, . . . , Ca+\ have to be provided /or each cache set to track the reuse behavior 
of cache lines. If, on a cache access, the requested cache line resides on position p 
of the LRU stack, counter Cp of the corresponding cache set is increased by one. If 
the cache access results in a miss, i.e. if the cache line has no corresponding entry 
on the LRU stack (and therefore the cache line does not reside in the cache), then 
counter Ca+\ is increased. This procedure leads to a so-called stack distance profile, 
as it is depicted in Fig. 2. The stack distance profile characterizes the positions of 
cache lines on the LRU stack when accessing cache data. 

Given a stack distance profile, the total number of accesses a to a specific cache 
set can simply be determined by summing up all C; according to 



a-i-l 
a — 2^ Ci 

i = \ 

and the cache miss rate Pm can be calculated by 

71 Ca+l 






Ci 



For caches with lower associativity a' , the miss rate can be computed by 



Pm(a)^ 



Ea + i ^ 
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ff+1 ^ 
1 ^i 



E? 



(1) 



(2) 



(3) 



Chandra et al. exploit this equation to predict the cache miss rate under cache shar- 
ing. Let c be the size of a cache and let c'^ be the effective cache size that is available 
for thread x under cache sharing and let S be the number of cache sets, then Chan- 
dra et al. estimate the effective associativity a' of a task when sharing the cache with 
another task according to 



S 



(4) 
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Fig. 3 Relationship between ^gq (41) 
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For their FOA model, Chandra et al. calculate the effective cache size by 

Ea+l ^ 
„ _ ' = 1 ''^ 



EN v^ff+l ^ 
3>=1 Z^i = l *-'■ 



(5) 



For the SDC model, Chandra et al. create a new stack distance profile by merging 
individual stack distance profiles to one profile and determine the effective cache 
space for each thread "proportionally to the number of stack distance counters that 
are included in the merged profile" [1]. The shaded region in Fig. 2 shows how the 
effective cache size is reduced by cache sharing. While the FOA and the SDC model 
both are heuristic models, Chandra et al. also developed an inductive probability 
model that is based on circular sequences (cseq) rather than on stack distances and 
is reported to deliver more accurate results than the FOA and SDC model. 



2.3 Chandra et al.'s Circular Sequence Based Prob Method 

Chandra et al.'s Prob method is based on circular sequences. Circular sequences are 
an extension to stack distances in that they do not only take into account the number 
of accesses to the different positions on the LRU stack, but also the number of cache 
accesses between accesses to equal positions on the LRU stack. 

Therefore, Chandra et al. define a sequence seq^ (dx , Hjt) as "a series of n ^ cache 
accesses to d^ distinct line addresses by thread x, where all the accesses map to the 
same cache set" [1] and a circular sequence cseq(dx , rix) as a sequence seq^(dx,nx) 
"where the first and the last accesses are to the same line and there are no other 
accesses to that address" [1]. Circular sequences can be regarded as stack distances 
that have each counter C augmented with an additional vector n to hold a histogram 
of accesses for each distance. Figure 3 illustrates the relationship between sequences 
and circular sequences when accessing cache lines A, B, C and D. 

For their circular sequence based Prob model, Chandra et al. compute the number 
of cache misses nix for a thread x when sharing the cache with an additional thread 
y by adding to the stand-alone cache misses Ca+i the values of the other counters 
Ci ■ ■ .Ca, each multiplied with the probability that the corresponding circular se- 
quences cseq(dx,nx) will become a miss, where n^ corresponds to the estimated 
mean n for a specific d: 

a 

rrix = Ca+i + ^ Pm {cseq^(dx,nx)) x Q^. . (6) 
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Chandra et al. calculate the probability that the circular sequence cseqidx^n^) 
will become a miss by summing up the probabilities that there are sequences 
seqyidy, E{ny)) in thread y with a — d^ -\- \ < d^, < E(ny), where E(ny) repre- 
sents the expected value of n in the thread y: 



E(n,.) 



Pm {cseq^ (djc,n.c))^ ^ P {seqy (dy ,£(«,)))■ 



(7) 



E{ny) is estimated by scaling rix proportionally to the ratio of accesses of y and x: 



E(ny) : 






■^a+l 



Cl 



(8) 



The probability of sequences P{seqy(dy, Eiriy))), in short P{seq{d, «)), is calcu 
lated recursively according to 

' Ef=l P(cseq(.i, *)) ■ P(seq(S, v - 1)) 



P{seq(S,v)) 



+ il-EtlP(cseq(i,*))) 
X P(seqiS-l,v- 1)) 
P(cseq(l,*))-P(seq(l,v-l)) 

X P(seq{8- \,&- 1)) 



if V > ^ > 1 

if V > t/ = 1 (9) 



\fv — d>\ 
11 ifv^d = l. 

The asterisk (*) in cseq(i, *) denotes all possible values. 



3 Setvector Based Cache Contention Prediction 

In this section, I describe the Setvector method. First, I present the algorithm to 
obtain Setvectors. In a second step, I show how Setvectors can be used to predict 
cache contention. 



3.1 Generating Setvectors 



Setvectors are composed of cache set access frequencies a and the number of dif- 
ferent cache lines d referenced within a specific amount of time, for example an 
operating system's timeslice. For this contribution, I collect one Setvector for ev- 
ery interval of 2^" instructions. According to the proposal in [9] where I presented 
Setvectors to predict L2 cache performance of stand-alone applications, I assume an 
L2 cache with 32 bit address length that uses b bits to code the byte offset, s bits to 
code the selection of the cache set and k — 32 — s — b bits to code the key that has 
to be compared to the tags stored in the tag RAM. The Setvectors are gathered as 
follows. For every interval / of 2^*^ instructions do: 



Setvectors - An Efficient Method to Predict Cache Contention 251 

• First, set the 1 x 2' sized vectors a and d to 0. 

• Second, for every memory reference in the current interval, do: 

- Extract the set number o from the address, e.g. by shifting the address k bits to 
the left and then unsigned-shifting the result k + b bits to the right. 

- Extract the key from the address, e.g. by unsigned shifting the address s + b 
bits to the right. 

- Increase a[cr]. 

- In the list of the given set, determine whether the given key is already present. 

- If the key is already present, do nothing and proceed with the next address. 

- If the key is not in the list yet, add the key and increase d[CT]. 

This ends up with two 1 x 2'' dimensional vectors a and d. At index a, a holds 
the number of references to set ct and d holds the number of memory references 
that map to set a , but provide a different key. 

• In a third step, subtract the cache associativity a from each element in d and store 
the result in d'. If the result gets negative, store instead. 

• In a forth step, multiply each element of a with the corresponding element in d' 
and store the result in the 1 x 2' dimensional Setvector s, . 

• Finally, add S; as the i'th column of matrix S that holds in each column ; the 
Setvector for interval i . 

Process next interval. 



3.2 Predicting Cache Contention with Setvectors 

The compatibility of two threads for a time interval i can easily be predicted by just 
extracting s;^ from Sjc and s, from Sy and calculating the dot product s,\^. ■ s,^ of the 
Setvectors in order to obtain a single value. A low valued dotproduct implies a good 
match of the applications, a high dotproduct value suggests a bad match, i.e. a high 
level of cache interference resulting in many cache misses. 

The dotproducts do not have any specific meaning like number of additional 
cache misses, as it is the case with Chandra's circular sequence based method. How- 
ever, comparing the dotproducts of several thread combinations in relation to each 
other has been proven to be an effective way to predict which threads make a better 
match and which threads do not. 



4 Evaluating Cache Contention Prediction Techniques - 
Simulation Setup 

In order to prove the effectiveness of the Setvector method with its relative compar- 
ison of dotproducts, I compared it to Settle's Activity vector method and to Chan- 
dra's circular sequence based method. I refrained from additionally comparing the 
Setvector method to Chandra's stack distance based method, as Chandra already 
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Table 1 Parameterization of the MCCCSim simulator 

Parameter Private LI caclie Sliared L2 cache 

Size 32 k 2 MB 

Line size 128 Byte 128 Byte 

Associativity 2 way 8 way 

Hit time 1.0 ns 10.0 ns 

Miss time depends on L2 100.0 ns 

Replacement LRU LRU 



reported that the circular sequence based method outperformed the stack distance 
based methods - and the Setvector method showed nearly the same accuracy as the 
circular sequence based method. 

To compare and evaluate cache contention prediction techniques, I generated 
tracefiles with memory accesses representing 512 million instructions for each of 
the ten SPEC2006 benchmark programs astar, bzip2, gcc, gobmk, h264ref, hmmer, 
Ibm, mcf, mile and povray applying the Pin toolkit [4]. Of these ten programs, I exe- 
cuted every 45 pairwise combinations on the MCCCSim multicore cache contention 
simulator [10] that had been parameterized as it is shown in Table 1. 

For each program of each combination, I calculated the difference between the 
stand-alone memory access time and the memory access time when executed in co- 
schedule with the other application in units of picoseconds per instruction ipslinstr.). 
Figure 4 shows this additional penalty in the row Simulated Penalty for the astar 
application: If astar is co-scheduled with hmmer, then a penalty of 2.3 psiinstr. is 
introduced to the astar application. When co-scheduled with Ibm, astar suffers from 
a penalty of 134.7 ps/instr. As it can be seen from row Simulated Penalty in Fig. 4, 
I sorted the co-schedules according to the introduced penalty to yield a ranking such 
that a co-schedule with rank ( is favorable to a co-schedule with rank j > i. If two 
co-schedules were equally valued, I sorted them by name to avoid nondeterministic 
rankings. 

Similarly to the penalty ranking, I sorted the predictions I calculated for the 
Setvector method, Chandra et al.'s circular sequence method and Settle et al.'s Ac- 
tivity vector method. In Fig. 4, row Prediction shows an example of a prediction 
ranking. 

To evaluate the prediction methods, I introduce an evaluation technique I call 
mean ranking difference (MRD): Figure 4 shows that I calculate the absolute dif- 
ference between the rank determined by MCCCSim and the rank estimated by the 
prediction method for each combination. The results are summed up and divided by 
the total number of co-scheduled applications (9) to yield the average mean ranking 
distance (MRD), i.e. the mean number of ranks, a co-schedule's prediction differs 
from the values obtained from MCCCSim. 

I evaluated several variations of all three methods. Regarding Chandra et al.'s 
method, I was interested in comparing the predictions for the following variations: 
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Fig. 4 Determination of the mean ranking difference (MRD) for astar 



• Chandra cseq, chunkset: Prediction while applying only one circular sequence 
stack to a chunkset of instructions (i.e. interval of 2^° instructions). 

• Chandra cseq, Af(set): Prediction while applying a circular sequence stack to 
every cache set within an interval and measuring the memory access frequency 
on a per cache set basis. 

• Chandra cseq, Af(chunkset): Prediction while applying a circular sequence stack 
to every cache set within an interval without partitioning the memory access fre- 
quency on the cache sets, i.e. providing only one memory access frequency value 
per interval. 

Settle et al. stated that "the low order bits of the cache set component of a memory 
address are used to index the activity counter associated with each cache super set" 
[6]. However, I expected that the method would achieve better results when using 
the high order bits to index the activity counters since addresses with equal high 
order bits are mapped to equal cache sets. Therefore, I evaluated the Activity vector 
method for these two variants naming them high respectively low (cf. Table 2). 

Regarding the Setvector method, 1 was interested in analyzing the following vari- 
ations (cf. Table 2): 

• diff. X access: The Setvector method as presented in Sect. 3. 

• access: Utilizing only the access frequency. This way, the performance of the 
Activity vector method can be estimated for the case that the number of supersets 
reaches its maximum (i.e. the over all number of sets) and the activity expresses 
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Table 2 Mean ranking difference (MRD) for each benchmark and method 




Application 


Chandra 

cseq 

chunkset 


Chandra 

cseq 

Afiset) 


Chandra 

cseq 

Afichunsket) 


Activity 
low 


Activity 
high 


astar 


1.56 


0.89 


0.89 


3.56 


2.00 


bzip 


0.89 


0.44 


0.89 


2.67 


1.33 


gcc 


0.89 


0.67 


0.89 


3.11 


2.00 


gobmk 


0.67 


0.67 


0.44 


3.11 


3.33 


h264ref 


0.67 


0.67 


0.89 


2.67 


2.44 


hmmer 


0.89 


0.67 


1.11 


2.89 


2.44 


Ibm 


1.11 


0.67 


1.33 


4.00 


2.22 


mcf 


0.44 


0.22 


1.33 


3.11 


2.22 


mile 


0.67 


0.00 


0.44 


2.89 


3.11 


povray 


2.00 


0.89 


0.89 


2.67 


2.67 


average 


0.98 


0.58 


0.91 


3.07 


2.38 


Application 


Setvector 


Setvector 


Setvector 


Setvector 


Setvector 




dijf.x 
access 


access, 
add 


access, 
mul 


diff., add 


diff., mul 


astar 


0.67 


0.67 


0.44 


0.89 


0.89 


bzip 


0.67 


0.67 


0.22 


0.44 


0.89 


gcc 


0.89 


0.89 


0.67 


0.67 


0.67 


gobmk 


1.11 


1.33 


1.56 


0.89 


0.67 


h264ref 


0.44 


0.44 


0.44 


1.11 


1.11 


hmmer 


0.22 


0.22 


0.67 


0.89 


0.89 


Ibm 


1.33 


1.33 


1.56 


0.89 


0.44 


mcf 


0.00 


0.00 


0.89 


0.22 


0.22 


mile 


0.22 


0.44 


0.89 


0.22 


0.44 


povray 


0.44 


0.44 


0.22 


1.11 


1.11 


average 


0.60 


0.64 


0.76 


0.73 


0.73 



the number of accesses to a set and not just the one-bit information, whether or 
not a specific threshold has been reached. 

• diff: UtiUzing only the number of different cache lines that are mapped to the 
same cache set, i.e. ignoring any access frequency information. 

• add, mul: Combining the vectors of two threads by applying either elementwise 
addition or multiplication and calculating the average of the elements afterwards, 
rather than by applying the dot product. 
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5 Results 

Table 2 shows the accuracy of the evaluated methods and variations, Table 3 shows 
the execution time of the methods, subdivided into time that has to be spent offline 
(row setup), and time that has to be spend online (row pred.) when calculating the 
prediction for a specific combination. 

Table 2 shows that Chandra's circular sequence based method that utilizes the ac- 
cess frequency on &per set basis performs with the highest accuracy (MRD — 0.58). 
However, 676.83 picoseconds have to be spent per instruction (ps/instr.) on average 
to calculate the predictions with Chandra et al.'s method; this is about 6768 times 
as long as a prediction takes applying the Activity vector method (0.10 ps/instr.) 
and about 3981 times as long as a prediction takes applying the Setvector method 
(0. 17 ps/instr.). Although the Activity vector method performs quite fast, it shows a 
high error rate {MRD — 3.07 and MRD — 2.38 respectively). However, selecting the 
higher part of the set bits had been a good idea. Increasing the number of super sets 
to the number of sets and applying natural numbers to count the number of accesses 
to each set instead of using only a single bit per set significantly improves accuracy 
(MRD = 0.64, as seen from Setvector - access, add), but also increases prediction 
time (0.16). The Setvector method that applies both access frequency and number of 
accesses from different keys shows about the same prediction time (0.17 ps/instr.), 
but a slightly better accuracy (MRD — 0.60), that nearly matches that of the about 
4000 times slower performing circular sequence based method. 



6 Conclusion 

In this section, I presented state-of-the art methods to predict cache contention and 
proposed a new prediction method based on the calculation of so-called Setvectors. 
I simulated the additional memory access time introduced from cache contention 
during application co-scheduling and compared those values to the prediction meth- 
ods applying a new metric called MRD (mean ranking distance) that calculates 
the mean difference between the predicted and the simulated ranking. The results 
showed that the method introduced by Chandra et al. [1] might be the most accurate 
one, but it is nearly 4000 times slower than the proposed Setvector method, that 
achieves nearly the same accuracy {MRD — 0.60 instead of MRD — 0.58). 
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New Material Model for Describing Large 
Deformation of Pressure Sensitive Adhesive 



Kazuhisa Maeda, Shigenobu Okazawa, and Koji 
Nishiguchi 



Abstract A material model to describe large deformation of pressure sensitive ad- 
hesive (PSA) is presented. A relationship between stress and strain of PSA includes 
viscoelasticity and rubber-elasticity. Therefore, we propose the material model for 
describing viscoelasticity and rubber-elasticity and formulate the presented material 
model for finite element analysis. And we validate the present formulation by using 
one axis tensile calculation. 

Keywords Large deformation ■ Viscoelasticity ■ Rubber-elasticity • Pressure 
sensitive adhesive ■ Material model ■ Finite element method 



1 Introduction 

Pressure sensitive adhesive, which is called PSA afterwards, are industrially indis- 
pensable products in various fields. Its elastic modulus is about 10^ Pa at room tem- 
perature and indicates extremely low compared with other solid materials. There- 
fore, large deformation behavior can be observed in conventional PSA deforma- 
tion. 

Figure 1 shows the tensile stress-strain curve of PSA. As shown in Fig. 1, the 
stress increases exponentially for large strain zone. This behavior is called "rubber- 
elasticity" in this paper. PSA is generally considered to be a viscoelastic material. 
However, only viscoelasticity cannot describe practical behavior including rubber- 
elasticity of PSA consistently [1]. 

The generalized Maxwell model [2] is usually used for describing viscoelastic- 
ity. On the other hands, hyperelasticity is popular to simulate increase in stress [3]. 
However, there is a difficulty in use of hyperelasticity, because hyperelasticity in- 
dependent on time. In order to evaluate material parameters, hyperelasticity needs 
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Fig. 1 Stress strain curve of o.5 

PSA 
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time independent parameters with experimental data without stress relaxation. The 
aim of this study is the establishment of material model describing visco and rub- 
ber elasticity for PSA. The established material model can indicate rubber-elasticity 
without hyperelastic model. We formulate the above material model and its rate 
formulation for finite element analysis and then show the validation by using com- 
putational example. 



2 Material Model 

In this section, material model for PSA is described. After introducing general- 
ized Maxwell model, the model is extended to practical model including rubber- 
elasticity. 



2.1 Modification of Generalized Maxwell Model 



PSA indicates remarkable viscoelasticity at room temperature. The generalized 
Maxwell model for describing viscoelasticity is used in the present study. Figure 2 
shows generalized Maxwell model. Where / denotes unit number of the generalized 
Maxwell model. And E and rj are elastic modulus of spring and viscous coefficient 
of dashpot, respectively. 

The generalized Maxwell model of Fig. 2 cannot describe rubber-elastic behavior 
of PSA as shown in Fig. 1. The reason is that elastic modulus of the generalized 
Maxwell model is constant. Then, we propose evaluation of elastic modulus of the 
spring component. Figure 3 shows the modified generalized Maxwell model. This 
proposed model is called 'Advanced generalized Maxwell model". In the advanced 
generalized Maxwell model, elastic modulus is function of total strain. In addition, 
viscous coefficients of dashpot are function of strain to assume that relaxation time, 
r, is constant like the generalized Maxwell model. 
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Fig. 2 Generalized Maxwell 
model 
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Fig. 3 Advanced generalized 
Maxwell model 
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2.2 Elastic Moduli 



Elastic moduli for the advanced generalized Maxwell model are determined as the 
function of total strain. First, we measure stress relaxation behavior with various 
initial strains in order to investigate the strain dependency of an elastic modulus. 
For this experiment, standard acrylic PSA is used. Cylindrical PSA sample, whose 
cross-section is 2 mm^, is attached to a tensile machine so that the length of PSA is 
10 mm, and initial strain is given 100% by nominal strain. Then, the sample is elon- 
gated and keeps in the fixed strain. The stress relaxation curve is obtained by mea- 
suring the stress change at the measurement time. The relationship between nom- 
inal stress and nominal strain is changed into the relationship between true stress 
and true strain. Regression analysis is applied to the obtained curve using the stress 
relaxation formula of the generalized Maxwell model as shown in Bq. 1 , and we get 
the 5 sets of relaxation time, r' , and elastic modulus of the spring component, E' . 



a=^£'exp(^-^j 



(1) 



where a, E, r, and i denote true stress, elastic modulus of a spring component, 
relaxation time, and the unit number respectively. Then, the stress relaxation mea- 
surement with initial strain 200, 300, 400, 500, and 600% is measured, and we get 
the sets of the relaxation time and elastic modulus at each initial strain. 

Table 1 shows relaxation times and elastic moduli of each strain. The strains 
are converted into true strain. Elastic modulus E is depends on the strain and is 
considered about each case of r. The case of r = 100 [sec] is considered for an 
example. Figure 4 shows elastic modulus depends on strain. 
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Table 1 Relaxation times and elastic moduli of each strain 
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Relation 



True strain 



time [s] 


0.6931 


1.0986 


1.3863 


1.6094 


1.7918 


1.9459 


1.00 X 10° 


0.1541 


0.3637 


0.5694 


1.2057 


2.0546 


3.8663 


1.00 X 10' 


0.0326 


0.0832 


0.1368 


0.3548 


0.5107 


0.7711 


1.00 X 10^ 


0.0363 


0.0665 


0.1237 


0.1808 


0.2078 


0.3701 


1.00 X 10^ 


0.0250 


0.0519 


0.0761 


0.1040 


0.1549 


0.1522 


1.00 X 10'" 


0.0639 


0.1326 


0.2150 


0.3279 


0.4608 


0.5936 



Fig. 4 Relationship between 
elastic modulus and strain 




Considering relationship between elastic modulus and strain in Fig. 4. We de- 
cide to use the exponential function of strain as a function of an elastic modulus as 
follows. 



£' =A''exp(B'e) 
where A and B denotes material parameter, and e does strain. 



(2) 



3 Rate Form of Constitutive Equation 

Although the elastic modulus of the spring component of the advanced model is 
defined by Eq. 2, it is necessary to distinguish scalar, vector, and tensor strictly 
in the case of dealing with three dimensions. So, the elastic modulus of a spring 
component is replaced with Eq. 3. 

£' =A'exp(B'e) (3) 

where e denotes scalar of strain. And s is defined by Eq. 4 using the strain tensor, e. 

e — V« : «■ (4) 

It is assumed that viscoelasticity is in a deviatoric component in the model. So, 
a constitutive equation of deviatoric and volumetric component is respectively for- 
mulated, and then the constitutive equation of whole component is formulated. 
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3.1 Deviatoric Component 

The spring of /th unit is assumed to be an incompressible linear elastic material. 
The deviatoric stress tensor of /th spring unit is 

a"=2G'e'/''' (5) 

The superscript prime of a right shoulder shows a deviatoric component, e'^^' is 
a small strain tensor of the spring of /th unit. G is a shear elastic modulus, and the 
relation with E is as follows using Poisson ratio, v. 

The material time derivative of both sides of Eq. 5 gives 

^ = 2G'D- + ^^ (7) 

Dt Dt G' 

where D denotes rate of strain tensor. The left side of Eq. 7 does not have objectivity. 

So the constitutive equation is not objective. Therefore object stress rate is assumed 

^'^"^ .2G'D- + ^^ (8) 



Dt ;(,) Dt G' 

where subscript (*) means arbitrary objective stress rate. Equation 8 is used as a 
constitutive equation of a spring. 

Next, the dashpot of /th unit is considered. The shear viscous coefficient and the 
strain rate tensor of /th dashpot unit is expressed as r;' and D'^P'' respectively. And 
dashpot is assumed to be incompressible Newtonian fluid. The constitutive equation 
of /th dashpot unit is given as 

a"^2rj'D'^P'' (9) 

The model property insists that the strain tensor of /th unit can be assumed to be 
equal to the strain of the whole model. That is, e — s'^P'' + s^'P'' . The material time 
derivative of this equation gives 

From Eqs. 8, 9, 10, the constitutive equation of the /th unit's deviatoric compo- 
nent is derived to 

/r)rr''\ nrJ n-"' 

(11) 



(12) 



(Da"\ _ 


-C'DI^^V" 


V Dt )^,) 


Dt 2(1 + v')x' 


where v is Poisson ratio. Equation 3 can be changed to 




Dt Dt 


And Eq. 1 1 becomes 




(D(x"\ 


^rr'n i r' n'' "" 


V Dt )^^~ 


Dt 2(1 + v')t' 



(13) 
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Since the stress of the whole model is derived from summation of the stress of 
each unit, the constitutive equation of the whole model is expressed to 



Da" 
Dt 



(*) 



= ^(2G'D+5'^a" 



Dt 2{\ + v')x' 



(14) 



3.2 Volumetric Component 



Here it is assumed that the volumetric component is a compressible linear elastic 
material. Pressure, p, is given as 



p — —Ky tre 



where K^ is defined as follows 



^. = E 



/ = ! 



3(1 -2v') 



The material time derivative of Eq. 15 gives 
Dp DKy p 



Dt Dt K„ 



KArT) 



where 



DK, 
Dt 



E 

(=1 



1 DE' 

3(1 -2y') Dt 



Substituting Eq. 12 into Eq. 18 gives 
DK, 



Dt 



= E 

(=1 



B'E' De 

3(1 -2i;')^ 



Therefore, the constitutive equation of volumetric component is 



Dp^_P_ 

Dt Ky ^ 



E 



(=1 



B'E' De 

l>(\-2v')~Dt 



KyUD 



(15) 



(16) 



(17) 



(18) 



(19) 



(20) 



3.3 Whole Component 



Here the Jaumann rate is selected for objective stress rate. The material time deriva- 
tive of Cauchy stress and its Jaumann rate are related in Eq. 21. 



Da 



J 



U) 



Da 
l)t 



W(T-<rW 



(21) 
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where W denotes spin tensor, and the lower right (/) shows the Jaumann rate. Here 
the Cauchy stress is divided into deviatoric and volumetric components, 



a — a' — pi 
where I denotes 2nd unit tensor. Substituting Eq. 22 into Eq. 21 gives 

Dp, 



Da\ _(Da' 



Dt), 



V Dt 



Dt 



I 



(22) 



(23) 



U) \ ^' / (7) 

Therefore, substituting constitutive equation of deviatoric and volumetric compo- 
nent into Eq. 23, the constitutive equation of the whole component is. 



D<s'\ v^ f , ,De „. 



-E 



Dt 
B'E' 



2(1 + v')t' 
Ds 



3(1 -2i;') Dt 



K„ix\) 



(24) 



4 Explicit Finite Element Method 

The present study employs an explicit finite element method [4] to calculate the 
following computational example. The explicit finite element method is computa- 
tionally robust because of no iterations. 



4.1 Discrete Equilibrium Equation 



The equilibrium equation ignoring the body force is, 

da 



pa-. 



9x 



(25) 



where p is the material density, a is the spatial acceleration, and a is the Cauchy 
stress. 

We can derive the virtual work equation by multiplying both sides of Eq. 25 by 
the arbitrary virtual displacement 5u with the Gauss' divergence theorem of vol- 
ume V . 



/ pa- &\idV + 
Jv 



/ cr :(Se)dV^ / t ■ ^u 
Jv Jav 



dS 



(26) 



where t is the external surface force on the boundary area dV, and e is the linear 
strain as follows. 



1 



\dx) 



(27) 
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The discrete equilibrium equation can be derived using the finite element as fol- 
lows; 

Ma + Fint = Fext (28) 

where M is the mass matrix, Fint and Fgxt are the internal and external force vectors 
respectively. For the numerical integration of the isoparametric element in the plane 
strain state, the selective reduced integration method is used to avoid volumetric 
locking [5]. 



4.2 Central Difference Method 

To advance the time of the discrete equilibrium Eq. 28, we select the central differ- 
ence method. Here, Ar is the time increment from time r" to f "+^ The current time 
is f" and any properties of the material at time f""*"^ will be explicitly calculated with 
the central difference method. The material coordinate x at ?"+' is evaluated with 
the material velocity v at the central incremental time f"+'/^. 

x"+'=x"+v"+^''2^f (29) 

where the material velocity v at time f"+V2 jg 

v"+^/2^v"-'/2_^a"A? (30) 

and the spatial acceleration a at time f" is solved as follows with Eq. 28. 

a«=M-i(F;',.-Ff„,) (31) 

Equation 31 requires no solution of the simultaneous equations by using the di- 
agonal lumped mass matrix for M. 



5 Computational Results 

Here, in order to verify an above-mentioned technique, one axis tensile deformation 
of PSA is analyzed. 



5.1 Material Constants 

The material constants which must be defined in the constitutive equation are A' , 
B' and r' . These values are calculated from the experimental data of one axis elon- 
gation measurement. 

First, the method of one axis elongation measurement is explained. It is mea- 
sured at room temperature. Cylindrical PSA sample whose cross-section is 2 mm^ 
is attached to the tensile machine so that the length of PSA is 10 mm, the sample is 
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Relaxation times [s] 



A' [Pa] 



B' 



1.00 X 10° 
1.00 X 10' 
1.00 X 10^ 
1.00 X lO"* 
1.00 X lO*" 



1.28 X 10^ 
8.93 X 10' 
3.71 X 102 

1.30 X 10^ 

7.31 X 11 



1.17 
4.40 
3.67 
0.67 
3.85 



Fig. 5 Boundary conditions 




elongated with constant speed. The speed is 10, 50, 300 mm/min. Since the mea- 
surement data is nominal stress and nominal strain, the stress-strain curve is changed 
into true stress-true strain. 

The constitutive equation of the advanced model calculated by one dimension is 

da 
~dt 



^A^.MB'.)-T.{h-^'%y 



(32) 



Equation 32 is applied to the stress-strain curve with nonlinear least squares 
method, and the material constants are determined. Approximate curve is calculated 
so that all stress-strain curves with different three elongation speed are satisfied. The 
result is shown in Table 2. 



5.2 Comparison with Experiment 



In order to verify the proposed constitutive equation, computational results is com- 
pared with experiments. The stress-strain analysis of PSA is treated in order to ex- 
amine the quantity with experimental results. Figure 5 shows analysis model and 
boundary conditions with uniform deformation. The analysis object is cubic PSA 
whose length of one side is 1 cm. The number of elements of the material model 
as shown in Fig. 3 is set to 5, and material parameters use the values of Table 2. 
The density of PSA uses 1000 kg/m^. In order to assume near incompressibility, 
the Poisson ratio in all the elements of the material model is set to 0.49. 

The aim of this paper is to develop the constitutive equation of a PSA. So, the 
analysis is calculated by finite element analysis with one lattice. This analysis em- 
ploys the Lagrangian formulation, which is usually used for solid analysis. The La- 
grangian computational mesh is follows material. 

The tensile speed is 5, 50, 500 mm/min, which intentionally differ from the 
rates used at identifying the material parameters. Figure 6 shows the comparison 
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Fig. 6 Comparison between 
experiment and 

computational result in ,— 

tensile speed 5 mm/min ^ 
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Fig. 7 Comparison between 
experiment and 
computational result in 
tensile speed 50 mm/min 
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of an experimental and computational result at elongating speed 5 mm/min. The 
computational result is well consisted with the experiment. 

Figures 7 and 8 show the comparison of an experimental and computational re- 
sult at tensile speed 50 and 500 mm/min respectively. Both show good agreement 
between experimental and computational results. All results show that computa- 
tional results can describe increase in stress for rubber elasticity. Furthermore, they 
show that computational results can describe the rate dependability relevant to the 
viscoelasticity, which is another PSA's property. 



6 Conclusions and Future Works 



This paper has described the material model, which can describe the deformation of 
PSA. Our results indicate the following. 

1. The present model, advanced generalized Maxwell model, can describe visco 
and rubber elasticity. 
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Fig. 8 Comparison between 
experiment and 
computational result in 
tensile speed 500 mm/min 
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2. We have formulized the three-dimensional constitutive equation of the advanced 
generalized Maxwell model. 

3. We have validated the proposed advanced generalized Maxwell model with the 
one axis tensile analysis. 

The remained subjects to simulate practical PSA behavior with large deformation 
are as follows. 

1. The present analysis uses dynamic explicit method. Therefore, computational 
time step size is extremely small because of the requirement of the Courant con- 
dition. It is necessary to consider the solution method, which can use large time 
step size, for suitable analysis of PSA deformation. 

2. The highly distorted Lagrangian finite elements cannot retain numerical accu- 
racy. The present formulation should be extended to an Eulerian formulation [6, 
7], which is attractive for large deformation problem like PSA. 
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QoS Provisioning in EPON Systems with 
Interleaved Two Phase PoUing-Based DBA 



I-Shyan Hwang, Jhong-Yue Lee, and Zen-Der 
Shyu 



Abstract Ethernet Passive Optical Networks (EPONs) are designed to deliver mul- 
tiple services and applications, such as voice communications, standard (SDTV) 
and high-definition video (HDTV). To support these applications and their vari- 
ous requirements, EPONs require Quality-of-Service (QoS) mechanisms to be built 
in service. For this purpose, a scalable Interleaved Dynamic Bandwidth Alloca- 
tion (IDBA) mechanism for sharing the uplink bandwidth among optical network 
units (ONUs) is proposed in this paper. The modus operandi of IDBA is to di- 
vide the cycle time by partitioning the ONUs into two groups with some timing 
overlap to execute interleaved bandwidth allocation, which cooperates with Lim- 
ited Bandwidth Allocation (LBA), Excess Bandwidth Reallocation (EBR) and ac- 
curate prediction mechanism in EPONs. The proposed IDBA mechanism has two 
advantages, namely it eliminates the idle period problem in the traditional DBA 
mechanism, and guarantees QoS services by dynamically adjusting the bandwidth 
within the group of subscribers. This will not only support the differentiated ser- 
vices architecture but also offer various QoS levels. Simulation results obtained 
show that the proposed IDBA mechanism achieves desirable system performance 
relative to packet delay, jitter performance, throughput, ratio of packet loss and fair- 
ness. 

Keywords EPON ■ QoS ■ IDBA ■ System performance ■ EBR 



1 Introduction 

Due to the rapid and consistent increase in network traffic generated by domestic 
and small business users over the last few years, broadband access networks have 
become increasingly important. However, though various technologies have been 
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Fig. 1 Tree-based PON 
topology 



A 



OLT „ "^ \ • 




1:N 



Passive 
Splitter 



ONU,v 



used to provide broadband access to networks in the area known as the "last mile" 
[1], they cannot simultaneously upgrade the current access network whilst provid- 
ing a low-cost and high-speed solution for broadband access services. One possible 
solution is Ethernet Passive Optical Network (EPON) that has been discussed in 
IEEE 802. 3ah as one of the extensions of Gigabit-Ethernet [2]. The EPON archi- 
tecture, as shown in Fig. 1, consists of a centralized optical line terminal (OLT) 
and a number of splitters. It connects a group of associated optical network units 
(ONUs) over point-to-multipoint topologies to deliver broadband packets and re- 
duces costs relative to maintenance power. EPON also provides bi-directional trans- 
missions, i.e. downstream transmission from OLT to ONUs and upstream trans- 
mission from ONUs to OLT in sequence. To avoid data collision, the multi-point 
control protocol (MPCP) is introduced on EPON, and two media access control 
(MAC) messages: GATE and REPORT messages are included in MPCP proto- 
col [3]. 

In the traditional DBA scheme, the OLT will begin bandwidth allocation after 
collecting whole REPORT messages, resulting in the idle period problem (as shown 
in Fig. 2). To elaborate, the idle period is the sum of the computation time of DBA 
and the round trip time between OLT and each ONU. Hence, for the DBA scheme, 
reducing the idle period becomes one of the important issues to address in order to 
improve bandwidth utilization. Another problem with the DBA scheme is that the 
queue state is inconsistent due to packets that continue to arrive during this waiting 
time. To clarify this problem, refer Fig. 3 illustrates the waiting time, t2 — ti, which 
is the time elapsed from when the packets begin to arrive to before the start of 
data transmission. Consequently, packets that arrive during the waiting time have 
to be delayed to the next transmission cycle, potentially leading to longer delays. 
To address this, predictive schemes can be used so that traffic arrival during the 
waiting time is taken into consideration to avoid longer packet delay and network 
performance degradation. 

In this paper, we discuss the precision of an accurate traffic prediction mecha- 
nism, which is necessary to avoid over-estimation or under-estimation that can result 
in longer packet delays and degrade the network performance [4-7]. Although ex- 
haustive queue size prediction mechanism have been proposed (which can be credit- 
based [4, 8, 9], linear-based [4, 5, 10], proportion-based [5, 11], waited-based [12] 
or QoS-based [6-8, 13, 14]), these traffic prediction mechanism are unable to pro- 
vide feasible solutions for differentiated services and have not addressed the queue 
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size inconsistent problem. In this paper, the proposed traffic prediction mechanism 
supports differentiated services with their various requirements. 

Previous researches have suggested that the maximum of Tcyde is 1 ms [15, 16], 
which is set to meet the ITU-T recommendation, G.114, i.e. the delay for voice 
traffic in the access network to be set at 1.5 ms [17]. On one hand, making Tcyde too 
large will lead to longer packet delays for all Ethernet frames because a larger cycle 
time incurs a larger transmission window size and results in the ONU ineffectively 
holding the transmission channel. As a result, the backlogged traffic at the next ONU 
experiences longer packet delays. On the other hand, making Tcyde too small will 
result in more bandwidth being wasted by guard intervals and an increase in CPU 
processing load. 

From our previous studies [5], we noted that the idle period problem of IPACT 
can be resolved by using Early-DBA mechanism with Prediction-based Fair Exces- 
sive Bandwidth Reallocation Scheme (PFEBR), which includes the unstable degree 
list to provide more accurate predictions. However, the Early-DBA has a jitter per- 
formance problem, which is due to a change in the transmission orders of some 
ONUs. A scalable Interleaved Dynamic Bandwidth Allocation (IDBA) mechanism 
is proposed in this paper, which uses the concept of Early-DBA to resolve the idle 
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OLT 



ONUS 



S^ 1 transmission time S^ 2 transmission time S„+i jtransmission time S,j+i 2transmission time 



> REPORT Message 

> GATE Message 

Fig. 4 Operation of the proposed IDBA mechanism 

period problem. Nevertheless, the proposed IDBA does not change the granting or- 
der for ONUs to reduce the jitter problem. 

In proposed IDBA mechanism, shown in Fig. 4, the Tcyde is divided by halv- 
ing the ONUs. One part is the first subgroup {Group 1) of ONUs, which is denoted 
by 5'n,i transmission time for cycle n, and the other part is the second subgroup 
{Group!) of ONUs {Sn,2)- The subgroup S„+\^i upstream transmission period is 
calculated in the rath cycle. At Group! DBA time, the OUT performs the DBA com- 
putation for ONUs in subgroup Sn2- At this time, the OLT has granted the GATE 
message to ONUs in subgroup Sn+\,\_, so that the ONUs in subgroup S„+\\ can 
transmit upstream data during idle time while the OUT computes DBA for subgroup 
Sn+\,2 (Group2 DBA). ONUs in subgroup Sn+\,2 is allowed to transmit upstream 
data as soon as the last ONU in subgroup Sn+\,i finishes transmission. The OUT 
lets the DBA process execute the QoS-based prediction, the limit bandwidth allo- 
cation (LBA), and the excessive bandwidth reallocation (EBR) for each part. When 
the predicted bandwidth has been over-estimated, the unused bandwidth is simply 
reserved for the next part, and the total transmission time of two successive parts is 
limited in one Tcyde- The proposed IDBA has two contributions: one is eliminating 
the idle period problem in the traditional DBA mechanism, and the other is ensuring 
QoS services by dynamically adjusting the bandwidth between Sn,\ and Sn,2, which 
not only supports differentiated services architecture but also offers various levels 
of QoS. 

The rest of this paper is organized as follows. Section 2 presents the proposed in- 
terleaved DBA and QoS-based scheduling scheme. Simulation evaluations are pre- 
sented in Sect. 3, and conclusions are drawn in Sect. 4. 



2 Proposed Interleaved DBA Mechanism 

The Interleaved Dynamic Bandwidth Allocation (IDBA) mechanism is proposed to 
resolve the idle period problem and enhance the QoS for differentiated services by 
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Fig. 5 Flowchart of IDBA mechanism 



using prediction and EBR in the EPON system. The flowchart of the IDBA mech- 
anism is illustrated in Fig. 5, where Pnj is defined as the bandwidth request pre- 
diction of ONU/ for cycle n, SnjB^'" is defined as the minimum guaranteed band- 
width belonging to part i of ONU/ for cycle n, which can be calculated by the 
service level agreement (SLA), B^xcess the excess bandwidth which is calculated by 
the sum of the under exploited bandwidth of lightly-loaded ONUs and B remain, the 
unused bandwidth after excess bandwidth reallocation from heavily-loaded ONUs. 
When receiving whole REPORT messages from each ONU, the total available 
bandwidth can be calculated as (r x (T^'^j^^ — N x Tg) — N x 512), where r is the 
transmission speed of the EPON in bits per second, T^"jl is the maximum cycle 
time, A^ is the number of ONUs, Tg is the guard time and the control message 
length is 512 bits for the EPON system. Next, the QoS-based IDBA executes the 
prediction mechanism based on the current traffic status. The limited bandwidth 
allocation mechanism then compares the minimum guaranteed bandwidth with the 
predicted bandwidth of each ONU. If P„i < SnjB^'", then excessive bandwidth 
collection for lightly-loaded ONUs is executed, followed by excessive bandwidth 



276 I.-S. Hwang et al. 

reallocation mechanism for heavily-loaded ONUs. In the end, the unused bandwidth 
from the over-estimated bandwidth can be reserved for the next group of ONUs for 
DBA. Therefore, the IDBA can support QoS and enhance system performance for 
differential services and efficiently reallocates excessive bandwidth in EPON. The 
prediction mechanism of IDBA is described in Sect. 2.1, LBA mechanism with 
fairness in Sect. 2.2 and EBR mechanism in Sect. 2.3 respectively. 

Initially, the available bandwidth for part one, Sn.iBavaiiabie, and part two, 
Sn.iB available^ fof cycle n. Can be calculated by applying Eqs. 1 and 2. 

/ iv \ 

S„ J Bavaiiabie — total available bandwidth X I I — N 5'„,2Vt'/), (1) 

\ j=N/2+l / 

I Nil X 

Sn2Bavaiiabie — total available bandwidth x j 1 — > S„jWj ), (2) 

where Wj is the weight assigned to each ONU^ based on its SLA. 



2.1 Interleaved DBA Prediction Mechanism 

In relation to resolving queue variation between waiting times and reducing the 
packet delay, the prediction mechanism of IDBA takes differential traffic character- 
istic into account to enhance the prediction accuracy for each ONU. In this paper, 
we divide traffic data into three priority classes, EF, AF, and BE by the definition 
of Differentiated Services [4, 18]. To achieve better performance for a time-critical 
application, for instance constant bit rate (CBR) for EF traffic and non-busty traffic 
mode, bandwidth should be assigned to the ONUs according to the rate of these 
applications. Therefore, the proposed prediction mechanism assigns the CBR band- 
width to EF traffic as it multiplies the previous request of EF by one plus the pro- 
portion of waiting time, T„aitmg.j, and cycle time, Tcyciej, for ONUj. Moreover, 
the traffic characteristic of AF and BE are variable bite rate (VBR) and busty traf- 
fic mode, and the proposed prediction mechanisms of AF and BE traffic compare 
the difference between the requested transmission window at the present cycle and a 
mean value requested transmission window of historical cycles. The predicted value 
of bandwidth requirements for differentiated traffic is expressed in Eq. 3, where R^ • 

represents bandwidth request of each traffic type of ONU/ in cycle n, and hJ is the 
average bandwidth requirements of the history cycle of each traffic type of ONU/, 
where T e{EF,AF,BE}. 



-fi,! — /■ ^ V^ ~r ^waiting, j I ^cycle,j) 



pAF ^ i^AF _ fjAF (3) 

pBE _ nBE jtBE 
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After the traffic forecast value is calculated in each ONU, the prediction mecha- 
nism can derive P^ ^ which represents the prediction value of each ONU/ in cycle n, 
where T e {EF,AF, BE). For AF and BE traffics, if Pj : > 0, the demand tends to 
increase gradually and so we update the forecast value to obtain the new bandwidth 
requirements. Otherwise, if P^ ■ < 0, we do not update. 



2.2 Fair Limited Bandwidth Allocation Mechanism 

During dynamic allocation, the allocated timeslot will be adapted to the requested 
bandwidth. To prevent the allocation of excessive bandwidth (which can result in 
wasted bandwidth) or not enough bandwidth (which can increase packet delay), the 
proposed LBA is set as S„jGj — min(P„_y, SnjB^'"), where SnjG j is the granted 
bandwidth timeslot in GATE message for ONU/ (which belongs to the ith part in 
the nth cycle), P„j is the predicted value of bandwidth requirements for ONUj, 
and Snj B^'" is the minimum guaranteed bandwidth of ONUj that will equalize the 
bandwidth for each ONUs in part /. If P„j < SnjB^'", the limited granted band- 
width from the OLT is the same as the predicted bandwidth; otherwise, the grant for 
the ONU/ equals S„ iB^'"- The proposed LBA not only solves the problem of an 
ONU with heavy traffic load monopolizing the upstream channel, but also supports 
the priority servicing of differentiated services to guarantee QoS. The SnjB^'" and 

SnjG^ , where T e {EF,AF, BE], in GATE message based on each traffic class are 
described as follows: 

SnjBf" = SnjBavailablelNumbers of 5'„,,ONU, (4) 



Y = mm{Pf^,S„jBf'') 

AF rTnkr,{ jy^F c nMui C f^EF 



Snj Of = miniPf- ,Snj Bf" - S„j 0^,,+,) (5) 

r f-^BE c T>Min o f^EF o f^AF 



2.3 Excessive Bandwidth Reallocation Mechanism 

After LBA grants all bandwidth timeslot to the active ONU/, lightly loaded ONUs 
with bandwidth requirements less than the Snj b¥'" may still be present. The sum of 
the under utilized bandwidth of lightly-loaded ONUs, excessive bandwidth (B^^cess) 
[5, 6], can be expressed as Eq. 6: 

Bexcess ^ / X ^njB j ~PnJ)i (o) 

where Sn jB'^'" > Pnj, L is the set of lightly-loaded ONUs and y is a lightly-loaded 
ONU in L. 
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In the proposed EBR mechanism, B^xcess is redistributed among the heavily- 
loaded ONUs. A heavily-loaded ONU obtains an additional bandwidth based on 
the EBR mechanism. After EBR, if the B^j^cess is larger than the sum of the band- 
width request among the heavily-loaded ONUs, then a remaining available band- 
width, B remain, which Can be retained to the next excessive bandwidth collection to 
enhance the bandwidth efficiency in the next cycle. The Bremain must be restricted 
at half cycle time to avoid the piling up of unused available bandwidth, which can 
result in unfair resource distribution. Bremain is expressed in Eq. 7 as follows: 

^remain -- ti excess / ^ \^n,] ^nj'^j), \') 

where SnjB¥'" < P„j, H is the set of heavily-loaded ONUs and j is a heavily- 
loaded ONU in // . 

The proposed EBR mechanism in the IDBA can provide fairness to EBR based 
on the guaranteed bandwidth rather than the requested bandwidth, with no partial- 
ity and increase in bandwidth utilization. Moreover, the EBR mechanism not only 
alleviates the unfairness problem but also supports QoS by fairly distributing in a 
priority manner to improve LBA scheduling and enhance the traffic class. 



3 Performance Evaluation 

In this section, comparisons are performed using the PFEBR scheme, IDBA_Fixed 
without LBA, EBR and prediction, IDBA_EBR without prediction mechanism, 
IDBA_EBR_Pre incorporated QoS-based prediction with EBR, hybrid double- 
phase polling algorithm (DPA) [8], with respect to end-to-end delay, throughput, 
jitter performance, ratio of packet loss and fairness. The system model is set up 
within the OPNET simulator with one OLT and 32 ONUs. The simulation scenario 
is summarized in Table 1. For the traffic model considered, an extensive study has 
shown that most network traffic can be characterized by self-similarity and long- 
range dependence (LRD) [19]. In order to simulate the effect of high priority traffic, 
the proportion of traffic profile is analyzed by simulating the three significant sce- 
narios in (EF, AF, and BE) with (20%, 40%, 40%), (40%, 30%, 30%), and (60%, 
20%, 20%), respectively [6, 20]. 



3.1 End-to-End Packet Delay 

Figure 6 compares the mean end-to-end packet delay and EF traffic classes with 
end-to-end delay vs. different traffic loads for PFEBR, IDBA_Fixed, IDBA_EBR, 
IDBA_EBR_Pre and hybrid DPA. The simulation results obtained show that hy- 
brid DPA has a higher average end-to-end packet delay than IDBA when the traffic 
load exceeds 60%. However, the mean end-to-end packet delay of IDBA_Fixed and 
IDBA_EBR_Pre increased when traffic load exceeded 70% while for IDBA_EBR, 
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Table 1 Simulation scenario 



Number of ONUs in the system 

Upstream/downstream link capacity 

OLT-ONU distance (uniform) 

Buffer size 

Maximum transmission cycle time 

Guard time 

Computation time of DBA 

Control message length 



- PFEBR _244 

- PFEBR_433 
PFEBR_622 



- IDBA_Fixed_244 

- IDBA_Fixed_433 

- IDBA_Fixed_622 



-|DBA_EBF!_244 
-IDBA_EBR_433 
-|DBA_EBR_622 
2 I- 



■ Hybrid DPA_244 

■ Hybrid DPA_433 
- Hybrid DPA_622 



32 

1 Gbps 
10-20 km 
10 MB 
1 ms 
5 (js 
10|JS 
64 bytes 



- IDBA_EBR_Pre_244 

- iDBA_EBR_Pre_433 

- iDBA_EBR_Pre_622 




10% 20% 30% 40% 50% 60% 70% 80% 90% 
Traffic load (%) 

(a) 



10% 20% 30% 40% 50% 60% 70% 80% 90% 
Traffic load (%) 

(b) 



Fig. 6 (a) Average end-to-end packet delay and (b) EF end-to-end delay vs. different traffic loads 
for PFEBR, IDBA^Fixed, IDBAJBR, IDBA_EBR_Pre and hybrid DPA 



it increased when traffic load exceeded 80% regardless of the scenario. One possi- 
ble reason is that the idle period problem can be resolved by IDBA mechanism and 
that the excess bandwidth can be reallocated effectively by using the EBR mecha- 
nism. Figure 6(a) shows that the IDBA mechanism has better performance in terms 
of mean end-to-end packet delay when the ratio of EF traffic increases. One possi- 
ble reason is that the network environment will converge to a stable state when the 
proportion of CBR traffic is higher. Figure 6(b) shows that the IDBA_EBR_Pre can 
guarantee the bandwidth for EF traffic class, and result in lower EF end-to-end de- 
lay for each ONU. The IDBA mechanism meets the ITU-T recommendation G.l 14 
that specifies the delay for voice traffic in the access network at 1.5 ms [17]. The 
end-to-end delay of EF traffic in the proposed method can be guaranteed regardless 
of the proportion of EF traffic. 
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Fig. 7 (a) Mean system throughput and (b) EF traffic throughput vs. different traffic loads for 
PFEBR, IDBA^Fixed, IDBA_EBR, IDBA_EBR_Pre and hybrid DPA 

3.2 System Throughput 

Figure 7 shows the mean system throughput against different traffic loads for the 
PFEBR, IDBA_Fixed, IDBA_EBR, IDBA_EBR_Pre and hybrid DPA. Figure 7(a) 
shows that the proposed IDBA mechanism outperforms hybrid DPA in mean system 
throughput, and this is because interleaved transmissions can eliminate the problem 
of idle time in traditional DBA and support efficient EBR as well as the interleaved 
remain compensation mechanism. In this case, the IDBA_EBR has the best mean 
throughput performance due to effective excess bandwidth reallocation and the in- 
terleaved remain compensation mechanism. As for the EF throughput, as shown in 
Fig. 7(b), the proposed IDBA_EBR_Pre mechanism outperforms the other mecha- 
nisms because the EF traffic obtains additional bandwidth by using QoS-based pre- 
diction mechanism which can enhance high priority traffic adaptively for different 
traffic proportions. 



3.3 EF Jitter and Packet Loss Ratio 



Figure 8 shows the comparison of delay variance of EF class and packet loss against 
different traffic loads among PFEBR, IDBA_Fixed, IDBA_EBR, lDBA_EBR_Pre 
and hybrid DPA, respectively. In the proposed IDBA mechanism, we can see that 
the EF jitter of PFEBR can be improved by using IDBA, especially IDBA_Fixed. 
The reason is that the transmission order of each ONU is sequential and that PFEBR 
changes the transmission order of ONUs. Figure 8(b) shows that the hybrid DPA be- 
gins to have packet loss when the traffic load exceeds 70% in every scenario due to 
over allocation of the requested bandwidth to ONUs [21] for the EBR mechanism 
in hybrid DPA. This is termed the redundant bandwidth problem [5] to decrease 
overall system throughput. IDBA_Fixed starts to have packet loss when traffic load 
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Fig. 8 (a) EF jitter and (b) average packet loss ratio vs. different traffic loads for PFEBR, 
IDBA_Fixed, IDBA_EBR, IDBA_EBR_Pre and hybrid DPA 
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exceeds 80% in every scenario because lake an effective excess bandwidth realloca- 
tion mechanism. Furthermore, IDBA integrates the EBR and remaining bandwidth 
compensation mechanism to improve bandwidth utilization that prevents packet loss 
build up in high traffic load for each scenario. 



3.4 Fairness 



Figure 9 shows the comparison of fairness against different traffic loads among 
PFEBR, IDBA_Fixed, IDBA_EBR, lDBA_EBR_Pre and hybrid DPA, respectively. 
Recently, fairness and QoS on DBA schemes have become important issues, which 
we are also evaluating. The fairness index / (0 ^ / ^ 1) has been addressed [22] 
which is defined as Eq. 8 

/=(EG[,]J /nJ2gI,, (8) 
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where A'^ is the total number of ONUs and G[,] is the granted bandwidth of ONU,. 
Jain's fairness index /, ranging from to 1, becomes 1 when all ONUs have the 
same amount of bandwidth allocated by the OLT. Simulation results show that Jain's 
fairness index/of IDBA is better than hybrid DPA. IDBA_EBR_Pre has the best 
fairness performance, where the average Jain's fairness index / is about 0.9. One 
possible reason is that the proposed IDBA_EBR_Pre utilizes the idle period and 
remaining bandwidth by performing DBA computation for fair bandwidth alloca- 
tion. The fairness of hybrid DPA begins to gradually vary from traffic load 40% 
to traffic load 70%. Two possible causes of this is that 1) the hybrid DPA changes 
the transmission mechanism between online polling and double phase polling and 
2) the EBR based on the requested bandwidth in hybrid DPA has some drawbacks, 
namely unfairness and excessive bandwidth allocated to ONUs over what has been 
requested. 



4 Conclusions 

In this study, important factors that can improve the performance of EPON are dis- 
cussed and evaluated. The IDBA mechanism executes an interleaved transmission 
process to automatically adjust cycle time to resolve the idle period problem for 
traditional DBA scheme, enhancing the system performance to reduce end-to-end 
packet delay and improving the throughput. Moreover, it not only accounts for the 
prediction for differential traffic characteristic but also allocates bandwidth for dif- 
ferential traffic adaptively and improves the utilization of bandwidth by using EBR 
and the remaining bandwidth compensation mechanism. The simulation results ob- 
tained show that the throughput of IDBA is better than that of PFEBR and hybrid 
DPA, especially in relation to EF traffic performance and average packet loss ra- 
tio. Furthermore, the proposed QoS-based prediction mechanism outperforms the 
hybrid DPA by lowering the EF packet delay. Finally, the IDBA can effectively im- 
prove the EF jitter problems faced by PFEBR. It makes use of excess bandwidth and 
remaining bandwidth to have higher system throughput, lower packet loss and end- 
to-end delay of each ONU. Future work includes finding out the optimal number of 
subgroups and having a more appropriate prediction mechanism to improve system 
performance. 
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The Game of « -Player Shove and Its Complexity 



Alessandro Cincotti 



Abstract Why are «-player games much more complex than two-player games? 
Is it much more difficult to cooperate or to compete? The game of «-player Shove 
is the n-player version of Shove, a two-player combinatorial game. In multi-player 
games, because of the possibility to form alliances, cooperation between players is a 
key-factor to determine the winning coalition and, as a consequence, « -player Shove 
played on a set of finite strips is VSV ACE -complete. 

Keywords Combinatorial games ■ Complexity ■ n -Player Shove 



1 Introduction 

The game of Shove is a combinatorial game defined in [1] and played on a finite 
strip of squares where each square is empty or occupied by a red or blue piece. Two 
players, called Red and Blue, move alternately. A piece of the current player's color 
and all pieces to its left are pushed one square to the left. Pieces may be shoved 
off the end of the strip. The first player unable to move is the loser. An example of 
Shove is shown in Fig. 1. 

n -Player Shove is the « -player version of Shove played on a set of finite strips 
of squares where each square is empty or occupied by a piece labeled by an integer 
;e{l, 2,. ..,«}. 

The first player pushes a piece labeled 1 and all pieces to its left one square to the 
left. Pieces may be shoved off the end of the strip. The second player pushes a piece 
labeled 2 and all pieces to its left one square to the left. The other players move in 
similar way. 
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Fig. 1 An example of Shove 
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Fig. 2 An example of queer 
game 



Players take turns making legal moves in cyclic fashion (1-st, 2-nd, . . . , n-th, 
1-st, 2-nd, . . .)• When one of the players has no legal play, that player leaves the 
game and the remaining n — 1 players continue playing in the same mutual order as 
before. The remaining player is the winner. 

We briefly recall the definition of queer game introduced by Propp [16]: 

Definition 1 A position in a three-player combinatorial game is called queer if no 
player can force a win. 

Such a definition is easily generalizable to n players: 

Definition 2 A position in an w-player combinatorial game is called queer if no 
player can force a win. 

In the game of n -player Shove is not always possible to determine the winner 
because of queer games. In the game shown in Fig. 2 two different scenario are 
possible: 

• If the first player move in the first strip, then the third player wins as shown in 
Fig. 3. 

• If the first player moves in the second strip, then the second player wins as shown 
in Fig. 4. 

In two player games [2, 11] players are in conflict to each other and coalitions 
are not allowed but in n-player games [3, 13, 14, 17], when the game is queer, only 
cooperation between players can guarantee a winning strategy, i.e., one player of 
the coalition is always able to make the last move. As a consequence, to establish 
whether or not a coalition has a winning strategy is a crucial point. 



The Game of n-Player Shove and Its Complexity 
Fig. 3 The third player wins 



287 



2 


Y 


2 




1 




^ 








2 







1 


3 


; 


1 


3 


i 


1 


t 



In previous works, we analyzed the complexity of w-player Hackenbush played 
on strings [4, 5], n-player map-coloring games [6-8], and n-player Toppling Domi- 
noes played on rows [9, 10]. In this paper we show that, in Shove, cooperation can 
be much more difficult than competition and, as a consequence, M-player Shove is 
VSVACS-complete. 



2 The Complexity of n -Player Shove 

In this section we show that the PSV AC £-coinp\ste problem of Quantified Boolean 
Formulas [12, 15], QBF for short, can be reduced by a polynomial time reduction 
to n-player Shove. 

Let (p = 3xiVx23x3 . . . Qxnif be an instance of QBF, where 2 is 3 for n odd and 
V otherwise, and i/'^ is a quantifier-free Boolean formula in conjunctive normal form 
where every clause has 3 distinct literals. We recall that QBF asks if there exists an 
assignment to the variables x\, xt,, . . . , X2\n/2'\-\ such that the formula evaluates to 
true. If n is the number of variables and k is the number of clauses in ^jr, then the 
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Fig. 4 The second player 

















A. Cincotti 


2 


1 


2 




Y 




3 






i 






^ 


1 


2 






3 






; 








1 


2 








It 





instance of « -player Shove will have 2n+k- 
as follows: 



- 2 players and 2n + 2 strips, organized 



For each variable x/, with 1 < ; < n, we add two new strips. Both strips have a 
piece labeled n + k + i, with n + k+l<n + k + i<2n + km the first square 
on the left and a piece labeled ; , with 1 < / < « in the second square on the left. 
In the first strip we add one piece for each clause that contains x,- and in the 
second strip we add one piece for each clause that contains xj. These pieces are 
labeled j, with n+l<j<n + k, and arranged in increasing order from left to 
right. 
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Fig. 5 An example of 
reduction 
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• The 2« + 1-th strip contains 4 pieces labeled 2n + k + I respectively in 
the first, (n + k + 2)-th, {n + k + 3)-th, and (n + k + 4)-th square of the 
strip. 

• The last strip contains n + k + 4 pieces labeled from left to right 2n + k + 2, 
n + l,...,2n,2n + l, ...,2n+k,2n + k + 2,2n + k + 2,2n+k + 2. 

Let us suppose that: 

• The first coalition is formed by Ln/2J + 1 players corresponding to the 2-nd, 4-th, 
. . ., 2[n/2J-th, and 2n + k + 1-th players, 

• The second coalition is formed by the remaining players. 

An example of reduction is shown in Fig. 5 where 

(p = 3xiVx23x3Vx4(C5 A C(, A C-j) 

and 



C5 = (xi VX2 VX3) 

Ce = (xi V X2 V X4) 

C7 = (xi V X3 V X4) 

The problem to determine the winning coalition is strictly connected to the prob- 
lem of QBF, as shown in the following theorem. 
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Theorem 1 Let G be a general instance of n -player Shove played on a set of 
strips. Then, to establish whether or not a given coalition has a winning strategy 
is a VSVACE-complete problem. 

Proof We show that it is possible to reduce every instance of QBF to a graph G 
representing an instance of n-player Shove. Previously we have described how to 
construct the instance of « -player Shove, therefore we just have to prove that QBF 
is satisfiable if and only if the second coalition has a winning strategy. 

If QBF is satisfiable, then there exists an assignment of Xj such that t^t is true 
with / e {1, 3, . . . , 2[n/2] — 1}. If x, is true, then the /-th player plays in the strip 
with the pieces corresponding to the clauses containing x, . If x, is false, then the 
(-th player plays in the strip with the pieces corresponding to the clauses containing 
x7. Every clause contains at least a true literal, therefore the i'-th player with ; e 
{n -\- \ , n + 2, . . . , n + k} can always push a piece in the strip corresponding to that 
literal. In this way, at the end of the first round, the 2n + k + 2-th player is able to 
shove the piece in the first square of the last strip and, at the end of the game, he/she 
will be able to make the last move. Therefore, the second coalition has a winning 
strategy. 

Conversely, let us suppose that the second coalition has a winning strategy. We 
observe that the 2n + A: + 1-th player is always able to make 3(« + fc) + 7 moves, 
therefore the 2n + k + 2-th player must be able to make the same number of moves 
in order to guarantee a winning strategy for the second coalition. As a consequence, 
the /-th player with i e {n + I, . . . ,2n + k} does not shove any piece in the last 
strip during the first round, i.e., every clause has at least one true literal and QBF is 
satisfiable. 

Therefore, to establish whether or not a coalition has a winning strategy in «- 
player Shove played on a set of strips is VSVACE-hard. 

To show that the problem is in VSPACE we present a polynomial-space re- 
cursive algorithm to determine which coalition has a winning strategy as shown in 
Fig. 6. 

Let us introduce some useful notations: 

• G — (V, E) is the graph representing an instance of n-player Shove; 

• Pi is the /-th player; 

• Co is the set of current players belonging to the first coalition; 

• C\ is the set of current players belonging to the second coalition; 

• coalition(p,) returns if pi e Co and 1 if pt e C\ ; 

• label(j;) returns the label of the piece v; 

• next(/7/) returns the player which has to play after pi ; 

• shove(G,D) returns the graph obtained from G after that the piece v has been 
shoved. 

Algorithm Check receives in input a graph G — {V, E), the two initial coalitions 
Co and C\ , and the player p, that has to move. It returns if the first coalition wins 
and 1 if the second coalition wins. 
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Fig. 6 A polynomial-space 
recursive algorithm 



Algorithm Check(G, Cq, Ci 


Pi) 




j ■(— coalition (p,); 






it$veV : label(y) = / then 




Cj^Ci\{pi]; 






If Cj = then 






return i — j; 






else 






return Check(G 


Co, 


Ci,next(p,)); 


end 






else 






for all u e V : label(D) = 


i do 


G' ■^— shove(G, 


J); 




ifCheck(G', Co, 


Ci, 


next(p,))=7 then 


return j ; 






end 






end 






return I - j; 






end 







Algorithm Check performs an exhaustive search until a winning strategy is 
found and its correctness can be easily proved by induction on the depth of the 
game tree. 

Algorithm Check is clearly in VSVACS because the number of nested recur- 
sive calls is at most \V\ and therefore the total space complexity is 0(|V|^). D 
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Modeling the Vestibular Nucleus 



Alexandru Codrean, Adrian Korodi, 
Toma-Leonida Dragomir, and Vlad Ceregan 



Abstract In recent years the vestibular-sympathetic reflex has received an increas- 
ing amount of attention due to the role it could play in the human organism in 
different types of scenarios. Despite this, quantitative models of this reflex mech- 
anism are still lacking. In this context, the current paper aims at taking a first step 
towards the modeling of the vestibular-sympathetic reflex by developing a model of 
the Vestibular Nucleus - the central part of the vestibular-sympathetic reflex. After 
a careful analysis of the limitations and uncertainties of the available experimental 
data from the literature, a three step modeling methodology for the Vestibular Nu- 
cleus is presented. After a description of the operations involved in each step, in the 
end, some preliminary results are shown. 

Keywords Vestibular nucleus ■ Interpolation ■ Firing rate ■ Vestibular-sympathetic 
reflex 



1 Introduction 

With the integration between neuroscience, engineering and medicine, new dis- 
coveries are emerging at a rapid research gradient, that evolve from fundamental 
knowledge of the role and behavior of different neural structures, to modeling as- 
pects towards a quantitative representation, and finally to applications that could 
be further used in clinical practice. In this direction, many studies are focusing on 
how the information is processed in different nervous centers of the human brain, 
and what kinds of signals carry this information to different organs (e.g. the heart). 
Usually, the physiological mechanisms through which these nervous control sig- 
nals influence the end-effector organs are called nervous reflex mechanisms, or in 
some cases nervous regulatory mechanisms. Thus, a nervous reflex mechanism is 
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associated with a specific nervous center (or more) and with the efferent organs 
involved. 

An important nervous center involved in several reflex mechanisms like the 
vestibular-spinal reflex, vestibular-ocular reflex, and the vestibular - sympathetic 
reflex is the Vestibular Nucleus (part of the Vestibular System- VS). Recent studies 
have focused on the role of the vestibular-sympathetic reflex, and its effect on the 
cardiovascular system (more specifically the heart) [1, 2]. However, assessing the 
influence of the vestibular-sympathetic reflex on the cardiovascular system (CVS) 
poses difficulties due to its interaction with the baroreflex, which is one of the main 
nervous regulatory mechanisms of the CVS on short-term. 

From these two nervous reflex mechanisms, only the baroreflex is understood at a 
quantitative level, and several models exist in the literature that have been validated 
with experimental data (e.g. [3]). For the vestibular-sympathetic reflex no current 
model exists in the literature. 

In order to obtain a more complete model for the nervous control of the CVS 
that could then be used in different clinical scenarios (e.g. orthostatic stress-head 
up tilt test/sitting to standing), a quantitative model of the vestibular sympathetic- 
reflex must be developed, that would be integrated with an existing baroreflex 
model. 

In this context, this paper aims in taking a first step in the direction of a quan- 
titative modeling of the vestibular-sympathetic reflex, by focusing on developing a 
model of the Vestibular Nucleus (the central part of the VS and the most important 
level of information processing of the vestibular-sympathetic reflex). 



2 Information Processing Through the Vestibular-Sympathetic 
Reflex 

2.1 The Main Components of the Vestibular-Sympathetic Reflex 

The interactions between the vestibular-sympathetic reflex mechanism and the 
baroreflex mechanism occur at the level of the Medulla Oblongata (MO) and, 
from an informational point of view, they are illustrated in the block diagram from 
Fig. 1. 

The nervous centers in the MO playing a role in cardiovascular regulation are: the 
Nucleus Tractus Solitarius (NTS), the Nucleus Ambiguous/Dorsal Motor Nucleus 
of the vagus (NA/DMNX), the Ventrolateral Medulla (VLM) and the Vestibular Nu- 
cleus (VN). The first level of information processing for the baroreflex is the NTS, 
while for the vestibular-sympathetic reflex, it's the VN. The NTS receives infor- 
mation from the baroreceptors («br signal) and from the cardiopulmonary receptors 
(wcp signal), whereas the VN receives information from the vestibular receptors VR 
- (.Jvr vector signal). The VR in turn detect a change in head acceleration - a signal. 
The second level of information processing for these two nervous reflex mechanisms 
is the VLM, and their effects are believed to be additive [1]. The output signal of 
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Fig. 1 Nervous centers in the 
MO involved in the nervous 
control of the CVS 
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the VLM is transmitted to the Sympathetic System, while the output signal of the 
NA/DMNX is transmitted to the Parasympathetic System. Afterwards, the Sympa- 
thetic and the Parasympathetic Systems transmit the nervous control signals to the 
CVS. 

The reference signal «j, coming from Higher Nervous Centers (HNC), provides a 
variable set-point for blood pressure. As the figure suggests, the HNC may influence 
the behavior of the VLM and the NA/DMNX, according to certain conditions (e.g. 
psychological factors). 

Finally, it can be concluded that the vestibular-sympathetic reflex is composed 
out of 4 main parts: VR, VN, VLM and the Sympathetic System. Regarding the 
VLM, not much is currently known, except that it's a possible convergence level be- 
tween the baroreflex and vestibular-sympathetic reflex and, as we mentioned earlier, 
that this convergence could be considered in a simplified manner as being additive. 
For the VR quite a few models are presented in the literature [4-6]. 

The VN has not yet been modeled in a quantitative manner, despite the fact that 
it could be regarded as the most important level of information processing for the 
vestibular-sympathetic reflex. Although it is considered to have many neurological 
functions, with strong interconnections with other nervous centers, only its con- 
tribution to the vestibular-sympathetic reflex is of interest for the current presen- 
tation. In this context, the most important step in the direction of a quantitative 
modeling of the vestibular-sympathetic reflex is to firstly determine a model for the 
VN. 

Before we start with the modeling aspects, and how we used the experimental 
results from [7] for developing a model for the VN, it's important to analyze what 
types of uncertainties can arise from neurophysiologic experimental data presented 
in the literature (e.g. [7]). 
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2.2 Assessing the Uncertainties and Limitations in 
Neurophysiologic Experimental Data 

A certified fact is that many systems were designed guided by an expert knowledge. 
Often, there is no other way to describe various functioning patterns and they are 
best expressed through information taken over from experiments. Therefore, math- 
ematical models of different nature can be obtained using the data provided by an 
expert. Generally, the steps that have to be followed in order to design a mathemat- 
ical model in the domain of biological systems, particularly the field of neurophys- 
iology, are strictly dependent on the information provided by an expert. The expert 
provides experimental data which can be processed and used to develop the model. 

Let's now consider the black box representation of a Neural System from Fig. 2a. 
The input signal is a current, which usually has a slow variation in time (with the 
exception when step input signals are used) in the range of nA. The output signal is a 
voltage, and it can either remain at a relative constant negative value (e.g. —70 mV), 
or voltage spikes can occur (called action potentials AP). When the output signal is 
at the constant negative level, the Neural System is considered to be inhibited, while 
when the output signal contains voltage spikes the System is excited. Thus, the main 
interest is to study the Neural System when output spikes or action potentials occur. 

It is well known that neurons generate voltage spikes at constant amplitude, and 
that the information received from the input current signal is practically modulated 
into the instantaneous spike rate of the output voltage spikes [8]. If the shape of the 
action potential is important from a cellular (micro) point of view in understanding 
the biochemical mechanisms involved, from a physiological (macro) point of view 
only the instantaneous rate is needed in order to describe the information process- 
ing that characterizes the Neural System. Figure 2b shows an alternative black box 
representation of the Neural System, where the output signal is a computed instan- 
taneous rate of the voltage spikes (also called instantaneous firing rate - IFR). 

In neurophysiology, experimental data obtained for individual neurons or a small 
group of neurons (which can be embedded in the above mentioned Neural System) 
is usually presented into two ways: 

- in the time domain - when the input current signal is usually a step or a ramp 
signal, and the output signal used is either voltage or the IFR; 

- in the frequency domain - when the input current signal is of sinusoidal form 
and the output signal used - the firing rate - is also of the same form, experimental 
frequency characteristics can be obtained. 

The shortcoming of time domain experiments is that the output signal used in 
most cases is voltage, which is not convenient for physiological modeling for the 
above mentioned reasons, and experiments in which the firing rate was used as out- 
put signal are very rare and with a limited number of scenarios (e.g. ramp input 
current for 3 slope values). So developing a physiological model based on time 
domain experimental data is rather difficult, unless one finds experiments for the 
Neural System using the firing rate as output in the literature. 
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The frequency domain experiments pose the advantage of using the firing rate 
as output signal, while frequency characteristics bring more insight into the overall 
dynamics of the system, but other types of drawbacks arise here. Firstly, the hypoth- 
esis that the Neural System is linear in the vicinity of the current operating regime 
has to be adopted. Only then frequency domain identification methods could be ap- 
plied in developing a model. Secondly, due to the technological limitations of the 
measurement equipment, the experiments can be done on a limited frequency range 
(e.g. the frequency range for the experiments done on the VN neurons is somewhere 
between 0.1 Hz and 50 Hz). And thirdly, the number of operating points for which 
such experiments are made is limited. If we consider the mean firing rate as an in- 
dex of the system's operating point, then for the VN neurons the mean firing rate 
range is usually between 10 spikes/sec and 100 spikes/sec (having in mind that the 
Neural System is capable of reaching mean firing rates of greater values than 100 
spikes/sec). This could all influence the model identification process at a certain 
degree varying from case to case. 

Despite the drawbacks of the frequency domain experiments, at present, this type 
of experimental data seems to provide the most quantitative information for our 
case study - the Vestibular Nucleus. Thus, we will further focus on using frequency 
domain experiments as a starting point in developing a model for the VN, and the 
time domain experiments will only be used later one for additional model validation. 

Now, that we have discussed about the limitations of the neurophysiologic exper- 
imental data found in the literature, we will move on to analyzing the uncertainties 
that may also appear. 

Uncertainty, from the control theory point of view, can be divided in two main 
categories: process uncertainty and signal uncertainty [9]. Process uncertainty re- 
lates to how the physical system (or process) handles the information received 
through the input signals and passes it to the output (a model is always a simpli- 
fied representation of the physical system). Signal uncertainty refers to "external" 
influences such as unmeasured disturbances or noise corrupting measurements and 
sometimes it can be even introduced involuntary through the adopted measurement 
technique (or experimental set-up for data acquisition). 

Let's now return to the Neural System from Fig. 2b, in the case when we want to 
use experimental frequency characteristics for identifying a model. We mentioned 
that the IFR output signal is not a physical signal, because it is actually computed 
based on the real physical output Voltage signal (Fig. 2a). The manner in which 
this calculus (or association) is done is not treated unanimously in the literature. 
Practically, the IFR signal consists of discrete points that correspond to the interval 
between two consecutive Voltage Spikes, and these points are eventually interpo- 
lated in order to obtain a continuous time signal. The main issue here is to what 
time moment these discrete points should be associated. 

For example, for our VN system, we found two main approaches for associating 
these time moments (shown in Fig. 3a and b) in the literature. Let's consider that 
for a train of five voltage spikes (corresponding to five time moments f i , . . . , fs) the 
IFR must be computed based on the time intervals between two consecutive voltage 
spikes {Ti = 40 ms, Tj — 50 ms, Ti, — T4 — 60 ms). Because we have four time 
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intervals, we will obtain four discrete values for the IFR signal: IFRi — l/Ti — 
25 spikes/s, IFR2 = 1/72 = 20 spikes/s, IFR3 = l/Ts = 16.7 spikes/s and IFR4 = 
1/74 = 16.7 spikes/s. 

Figure 3a shows the first approach used in [7] by du Lac et al., when each dis- 
crete point of the IFR is associated to the time moment of the spike at the beginning 
of each interval. So the four discrete values IFR1-IFR4 correspond to the time mo- 
ments ti-t4. 

Figure 3b shows the second approach used in [10] by Uno et al. and in [1 1] by Ris 
et al., when each discrete point of the IFR is associated to the time moment of the 
spike at the end of each interval. So the four discrete values IFR1-IFR4 correspond 
to the time moments tj-ts . 

Now if we compare the two methods, we can see in Fig. 3c that the first method 
introduces a lead component (anticipative behavior), while the second method in- 
troduces a lag component (a latency). Thus, the method in which the IFR signal is 
computed introduces a signal uncertainty. In order to take into account this uncer- 
tainty we must define an uncertainty area - in this example, this area is marked by 
curves (1) and (2), which refer to the two mentioned methods. The true evolution 
of the IFR might be somewhere in the middle (the curve with solid line), where the 
time values associated to the IFR values would be chosen at the middle of the T 
intervals. 

The errors introduced by this type of uncertainty can vary depending on the spec- 
trum of the input signal. This can be illustrated with the help of frequency charac- 
teristics, when the input signal and the output signal have the sinusoidal form from 
Fig. 2b. If we use the experimental frequency characteristics for the VN obtained in 
[7], we can calculate with approximation the frequency characteristics that would 
have been obtained by using the second method of computing the IFR signal. This 
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Fig. 3 Computing the IFR signal {(1) after [7], (2) after [10]) 



calculus is applied only to the phase-frequency characteristics, and is based on the 
following formula: 



ip2(a>i) — (p\ {(Oj) + A^(tt),), (Di e [O.Itt, 20jt] rad/s 
180 



(1) 



where «, represents a frequency value, (pi and <p2 are the phase characteristics with 
the first and second method, Aq) is a phase adjustment and AT,„ is the mean interval 
between two consecutive voltage spikes for a given output signal variation (which 
in this case is equal to the reciprocal of the mean firing rate - MFR). 

Figure 4 shows the experimental frequency characteristics for two MFRs ob- 
tained in [7] with method 1 (Amplitude-frequency characteristic - solid line, and 
Phase-frequency characteristic - dashed line), the Phase-frequency characteristics 
calculated using (1) that would correspond to the second method (also dashed line), 
and the Phase-characteristics that would be obtained if the middle of the inter-spike 
interval had been associated to the IFR discrete values (solid line). These last Phase- 
frequency characteristics have been obtained from the previous two with the for- 
mula: 

(PciMi) = (P2(c^i) + 0.5 ■ {(piicoi) - (P2{coi)), Oi e [0.27T, lOit] rad/s (2) 

Here, the uncertainty zone is marked by the Phase-frequency characteristics ob- 
tained with methods one and two (dashed lines). It can be stated that the errors due to 
signal uncertainties greatly increase with frequency. These results are in agreement 
with recent studies [12]. 

All in all, if we want to derive a mathematical model (for the VN in particular) 
from experimental data in the frequency domain, we must take into account the 
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Fig. 4 Experimental 
frequency characteristics for 
the VN neurons with 
uncertainty areas 
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uncertainty introduced by the method of calculating the IFR output signal. One way 
of doing this is by correcting the Phase-Frequency characteristics with the relations 
(1) and (2). It's also important to underline each time, the adopted hypothesis. 



2.3 A Modeling Methodology for the Vestibular Nucleus 



In this section we analyzed, with a focus on experimental frequency characteristics, 
how experimental data available in the literature can be used to develop a model of 
the VN. In order to provide a mathematical model for the channel s^^ -^ n^a of the 
VN (see Fig. 1), the experimental data from [7] was considered the most appropriate 
knowledge that could constitute a starting point for developing a proper model of the 
VN. For a more rigorous organization of the modeling process, we first established 
a three step methodology (3S-M) - Fig. 5. 

Through this methodology a fixed structure model with variable parameters able 
to reproduce the adaptive behavior of the VN at the input signal s^i will be obtained. 
This should be understood as follows: the physical signal Svr can be described us- 
ing two information carrier variables, the mean firing rate imfr and the stimulation 
input current Jvr, aggregated as a vector signal .?vr = [•^mfr ■Svr] (Fig- 1 and Fig. 6). 
The output variable of VN, the signal n^a, is obtained by processing iyr using a pro- 
cessing algorithm whose parameters are continuously adapted to the state of the VS 
represented by Smfi- In this context the model could be regarded as an adaptive one. 
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Fig. 5 Methodology for obtaining a complete model for the VN from experimental frequency 
characteristics of individual neurons 



In the first step of the 3S-M, based on the work from [13], a pair of interpolative 
Bode plots generators will be developed based on the few Bode plots determined ex- 
perimentally in [7] for different values of the mean firing rate (^mfr)- The generators 
will provide a nonparametric model in the form of amplitude-frequency characteris- 
tics ({|//|dB}c) and phase-frequency characteristics ({(/>h}c), calculated for different 
values of Smt. The ensemble of interpolators will approximate the frequency re- 
sponse of the VN in the range Smb e [10-56 spikes/s]. 

In the second step of the 3S-M, based on the work from [14], a continuous time 
model of the VN will be outlined. The continuous time model will describe the 
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Fig. 6 The integration of the complete VN model in the Human Organism 

input-output dependency s„ —>■ ny^, using p parameters that depend on ^mfr- For a 
given pair of Bode plots (i.e. a given value of s^t) an identification algorithm will 
be used to compute the p parameters. 

The aim of the third step of 3S-M (the work from [15]) will consist in an exten- 
sion of the model developed in Step 2 in a manner that will avoid the usage of the 
identification algorithm in current simulations. To do this, each of the p parame- 
ters will be determined as a function of s^nt by its own interpolator (p Parameter 
Interpolators or p-PI). This would permit the model to adapt the parameters to the 
current value of .Jnifr- 



3 Developing a Model of the Vestibular Nucleus 
3.1 Step One of the 3 S-M 



As we mentioned in the previous section, we will use the experimental frequency 
characteristics from [7] as starting point. These frequency characteristics were ob- 
tained for a small number of ^mfr values, by considering the following: the input 
signal Svr is of sinusoidal form in respect with the time t — So ■ sin(2jrMvr?) (with 
^0 being the signal's amplitude), the output signal Wyn is also a sinusoidal signal 
riynit) — NvnO ■ sin(27tnyyt + 0vn) (A^vnO is the signal's amplitude and the phase 0vn 
reflects the inertia that appears in the transmission of iyr through the VN). 

In order to obtain various frequency characteristics of the VN activity, two 
interpolative generators will be used (see Fig. 7). Both interpolative generators, 
GIF Amplitude and GIPphase, make use of two-dimensional lookup-tables that are con- 
stituted using the experimental data from [7]. The inputs of the interpolative tables 
are: the mean firing rate inifr and an instantaneous frequency signal Wyr- The outputs 
are the amplitude A'vnO and the phase 0vn- 

As it can be seen in Fig. 7, the first input of each generator is Smt, a signal 
with slow variation in time. Therefore, according to the experiments related in [7], 
next in the scenario, constant values will be considered for Smb, particularly: 10, 
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Fig. 8 Frequency characteristics obtained with linear interpolation using GIP 



20, 30, 40, 50 and 60 spikes/sec. Figure 8 illustrates the amplitude-frequency and 
phase-frequency characteristics generated with GIPAmpiitude and GIPphase obtained 
through linear interpolation. 

The ensemble of frequency characteristics highlights the hypothesis that the be- 
havior of the neurons may consist in jumps between characteristics corresponding to 
different MFRs. Moreover, it seems that the MFR of a neuron or a group of neurons 
can be correlated to its current operating point. 

Now, that we have established the experimental and interpolated frequency char- 
acteristics that we want to use in the modeling process, the uncertainties we men- 
tioned in Sect. 2 can be taken into account. Because in [7], du Lac used the first 
method in computing the IFR signal, before we proceed to step two of the 3S-M, 
we must obtain the corrected Phase-frequency characteristics (through which the 
uncertainties are taken into account). This can be easily done, for Phase-frequency 
characteristics corresponding to any value of the i'mfr (experimental or interpolated), 
by using relationships (1) and (2) presented in Sect. 2. Thus, the corrected Phase- 
frequency characteristics will follow the patterns shown in Fig. 4. 
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Fig. 9 The general structure of the VN model 



3.2 Steps Two and Three of the 3S-M 



In the second step of the 3S-M we focused on obtaining parametric models for the 
VN from the nonparametric models presented in step one. In doing so, we initially 
used the frequency characteristics for which the uncertainties were not take into 
account (the IFR signal was computed with method 1) - we'll consider these fre- 
quency characteristics as initial frequency characteristics. This work was presented 
in detail in [14]. There we arrived at a model with the structure from Fig. 9. 

Subsequently, the model was extended in the third step of the 3S-M [15] by 
adding the p-PI, and the results were validated in the frequency domain (with exper- 
imental and interpolated frequency characteristics). 

After all this, we continued by trying to develop a VN model from the corrected 
frequency characteristics (which take into account the uncertainties), by following 
the same three steps of the 3S-M. Step one is the same as for the first model. In 
step two, the model structure at which we arrived was simpler - (3). In step three 
we then extended the model in a manner similar to the one described for the first 
model, by adding the p-PI, and validating the results in the frequency domain with 
experimental and interpolated frequency characteristics. 

Now let's compare the results that we have obtained with the two VN complete 
models (one for the initial frequency characteristics and one for the corrected fre- 
quency characteristics). 



Hvn(^) = K ■ {b4s'^ + bis^ + b2S^ + bis + l) 
I {a^s +035 + a2S +flii' + l) 



(3) 



If until now we have only mentioned the results in the frequency domain, it's 
time we also discussed the behavior of the models in the time domain. Here lies 
the flaw of not taking into account the uncertainties mentioned. Because for the first 
model the uncertainties were not taken into account, that is the Phase-frequency 
characteristics were not corrected, the resulting model came out to be unstable (due 
to the large positive values of the phase on the frequency domain of interest). Thus, 
time domain simulations are not possible with the first model. On the other hand, 
when uncertainties are taking into account - for example with corrections of the 
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Phase-frequency characteristics in this case - the resulting model is stable and thus 
suitable for time domain simulations also. 



4 Conclusions 

The paper discusses the mains steps in developing a complete adaptive type model 
of the VN based on experimental frequency characteristics. Due to the uncertainties 
identified in the data presented in the literature, two actual models were developed 
for comparison reasons - one in which uncertainties were taken into account and 
one in which they were ignored. By comparing the results it was concluded that 
only the model in which the uncertainties were assessed is suitable for time domain 
simulations (due to issues of instability for the other model). Therefore, in our future 
work we will further use this model, and as a future development, such a model 
could be coupled with the output of the vestibular receptors. This will later allow 
for a complete model of the VS involved in the vestibular-sympathetic reflex to be 
obtained. 
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SPECT Lung Delineation 

A Complete 3D Approach 

Alex Wang and Hong Yan 



Abstract This is a review paper of our quest in developing and implementing an 
automated three-dimensional (3D) lung delineation method capable of handling sin- 
gle photon emission computed tomography (SPECT) lung scans with defective con- 
tours and/or varying maximum count value (MCV) and total count value (TCV). 
Six clinically significant datasets consisting of simulations and real subject scans 
are used consistently throughout our studies. We first develop a dynamic threshold- 
ing method which allows removal of background noise in a 3D volumetric fash- 
ion. Next, we implement 3D image processing techniques to enhance the SPECT 
lung contours. Finally, we develop 3D active contours to perform actual delineation. 
Quantitative validation using known-volume simulations and qualitative verification 
via experienced physicians are done to evaluate the methods. We achieve over 90% 
agreement on average throughout all six datasets. 

Keywords SPECT lungs ■ 3D active contours • Pulmonary embolism 



1 Introduction 

SPECT has long been favored over CT/MRI in diagnosing pulmonary embolism 
(PE) for its non-invasiveness and high sensitivity and specificity [1, 2]. Given the 
criticality of obtaining accurate lung contours for subsequent PE diagnosis [3, 4], 
common practice is to use a fixed percentage of MCV for thresholding [5-7]. How- 
ever, the accuracy of this approach is suspective to the presence of localized high 
deposition of radioactive agents known as "hotspots". Furthermore, the inherent 
fuzzy contour and 3D nature of SPECT images also pose a certain degree of com- 
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plication to traditional planar image processing techniques for contour extraction. 
This study serves as a review of the journey we have embarked upon in taking on 
the challenge of developing and implementing an automated method of delineating 
SPECT lung contours that is completely 3D in nature. 

First, through the statistical examination of 3D volumetric information underly- 
ing the SPECT images, we develop a dual exponential thresholding (DUET) method 
that is adaptive in nature [8]. However, the method is limited by the lack of spatial 
considerations, especially with SPECT lung scans that had defective contours due to 
low MCV/TCV. This limitation is overcome by combining DUET with traditional 
planar active contours [9]. While we deliver promising results in a subsequent study 
where we combine planar image processing techniques with 3D active contours 
[10], it is the follow-up studies where all aspects of the method is 3D in nature that 
conclude our quest for a truly 3D approach in delineating SPECT lung contours 
with great precision [11]. 



2 Dataset 



Two base datasets are repeatedly used throughout our studies to ensure data con- 
sistency and comparability of results - 90 SPECT ventilation scans of admitted 
hospital subjects and 350 Monte Carlo simulations. 

Each of the SPECT ventilation scans is acquired using standard protocols. After 
inhalation of approximately 40 MBq of 99mTc-Technegas, data is acquired using 
a dual-/triple-head gamma camera fitted with low-energy high-resolution collima- 
tors in the format of a 128 x 128 projection matrix for 120 projection angles. To- 
tal acquisition time for the ventilation scan is about eight minutes. The scans are 
then reconstructed using the OSEM block-iterative algorithm with eight subsets and 
four iterations. 3D Butterworth low-pass filtering with cut-off frequencies of 0.8 
cycles/cm at an order of 9.0 is applied without attenuation correction. The result- 
ing image set contains 128 slices measuring 128 x 128 pixels in size. Simulation 
wise, a base set of ten gated projections each consisting of 120 slices measuring 
128 X 128 pixels is generated using Monte Carlo simulation of photon emission 
from a phantom with a known volume of 61,660 voxels. To reach a statistically 
significant dataset size, we develop a mass generation method to derive the set of 
350 hotspot-free, normally ventilated simulations where each simulation contains 
128 slices also measuring 128x128 pixels in size [9]. We also develop a method 
which adds artificial hotspots into the simulations without impacting the statistical 
significance of the simulations [11]. 

Based on these primary SPECT datasets, six subsets shown in Fig. 1 are extracted 
according to clinically significant image characteristics. 
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Fig. 1 Dataset cliaracteristics 
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Dataset # 


Dataset Description 


1 


90 simulations with low MCV/TCV 


2 


50 simulations with normal MCV/TCV 


3 


30 simulations wKh hotspots 


4 


35 subject scans with low HCVH'CV 


5 


50 subject scans with normal MCV/TCV 


6 


25 subject scans with hotspots 




Fig. 2 Sample histograms of real subject SPECT scans. The X and Y axes represent pixel values 
and corresponding counts. The original values are colored black and the natural logarithmically 
transformed values are colored grey. The vertical, horizontal, and spotted regions roughly denote 
background, normal lung tissues, and high-value regions respectively 

3 Methods 
3.1 Duet 



As SPECT images contain 3D volumetric information, as well as noisy background 
and fuzzy contours, existing method of thresholding based on fixed percentages 
of MCV is found to be inaccurate. Hence, a dynamic adaptation would be more 
appropriate in performing preliminary background cleansing before application of 
sophisticated image segmentation methods. 

To cater for the 3D nature of the SPECT projections, a summation histogram 
of all the individual slices in each projection is created. Examination of these his- 
tograms reveals triple exponential function like patterns after application of natu- 
ral logarithmic transformation. There is a rapid component representing the back- 
ground, a near-horizontal component denoting most of the normally ventilated lung 
tissues, and a final rapid component signaling the high-value regions. See Fig. 2 for 
illustration. 

Given the above observation, we believe an adaptive threshold may be found 
as the intercept between the first two exponential functions once appropriate func- 
tions are fitted. The data range for exponential function fitting is first established. As 
our interests lie only in the first two components, a preliminary thresholding is per- 
formed to exclude the third component. A horizontal line h on the y-axis which most 
closely matches the second near-level component is identified via exhaustive linear 
least squares fitting. Let xi , X2, . . . , -*:„ represent the pixel values, and y\, y2, ■ ■ ■ , yn 
represent the corresponding histogram counts. Given the search range for h is be- 
tween zero and y„, for every incremental horizontal line hj, the sum of the squared 
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Fig. 3 Sample horizontal 
topographic representation of 
a subject SPECT scan. The 
dividing point identified in 
this case is x = 62 
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differences si is calculated using Eq. 1, where /z, with the minimal s, value is de- 
fined as h. The preliminary threshold tpre is then defined as the maximum x value 
with its corresponding y value above h; only data before tp^e is included for further 
analysis. 






yji 



(1) 



Once the data range is identified, Eqs. 2.1, 2.2 are fitted to the first two compo- 
nents respectively. As the search ranges for A, B, a, b, ki and k2 are restricted, they 
are again found via exhaustive linear least squares fitting. Finally, the threshold used 
for lung delineation is found as the intercept between the two functions. 



f{x)^Ae-"''+ki+h 



g(x) = Be 



-hx 



-k2 + h 



(2.1) 
(2.2) 



3.2 Active Contours 



Although we have shown that DUET alone achieves above 90% accuracy for 
SPECT images that are normally ventilated, its performance is less ideal when the 
images are low in MCV and/or TCV. Hence, while DUET serves as an excellent 
noise remover, a more sophisticated and rigorous contour extraction approach is 
needed. Having assessed various image segmentation methods, we decide upon ac- 
tive contours for its relatively fast processing time and effectiveness in handling 
fuzzy contours. 

Given a SPECT scan, after application of DUET to remove background noise, 
pre-processing tasks are completed. First, the lungs are divided into the left and the 
right sides respectively. This is done by creating a horizontal topographic repre- 
sentation of the scan characterized by two peaks representing the two lungs and a 
trough in between signaling the dividing point [9]. See Fig. 3 for illustration. 

Next, the starting region of interest (ROI) for active contours is identified. In 
our initial planar attempt, ROI is automatically defined as a rectangular box where 
its boundaries are the first non-zero valued pixels coming in from all four sides 
[9]. In the subsequent 3D attempt however, such simplistic definition of the starting 
ROI is not sufficient. Instead, after identifying the rectangular prism containing the 
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lung, an ellipsoidal mesh made up of triangular strips permutated in a Freudenthal 
triangulation-like fashion is constructed. Based on this 3D ROI, an n x m matrix 
M,^ is formed where n is the number of points in the ellipsoidal mesh, and m is the 
number of corresponding immediate neighboring points [11]. 

Finally, to assist subsequent implementation of active contours, a corresponding 
gradient image must be created. While standard approach is to apply planar Gaus- 
sian smoothing followed by Sobel edge detection on each slice, we apply 3D varia- 
tions of the Gaussian smoothing Gx,y,z and Sobel edge detectors Sx,y,z respectively 
shown in Eqs. 3, 4 to obtain a truly 3D gradient image. 
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The basic active contours equation is defined in Eq. 5 where E^c 



(4) 



Ecurv and 



Ei„,g represent the energy terms of continuity, smoothness and edge attraction re- 
spectively, while the three parameters a, p and y control their degree of sensitivity. 
The equation can be discretely defined as Eq. 6 where the three energy terms are 
described in Eqs. 6.1, 6.2, 6.3. The term d equals the average distance between the 
pairs (pi, pi+\) and V/ is the spatial gradient of the intensity image /, computed 
at each point. Given the focus here is to define a 3D approach, we will refer to the 
three energy terms as surface continuity, surface smoothness, and surface attraction 
here forth. 



e = / {a(s)Econt + P(s)Ecurv + y(s)Ei^f,) ds 



(5) 
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While surface attraction Ei^g remains the same as the traditional planar ap- 
proach, surface continuity and smoothness have to be modified to cater for 3D im- 
plementation. First, with surface continuity Ecom, note that the average distance d 
no longer refers to the overall average distance between all contour points and their 
corresponding neighboring points as in planar active contouring. Instead, it refers to 
the average distance between the current contour point and its neighboring points in 
a 3D context, redefining and expanding Eq. 6.1 to become Eqs. 7, 7.1. As for surface 
smoothness Ecun>, instead of calculating the squared sum of the distances between 
a contour point and its immediate pre- or post-neighboring points, the sum of the 
squared distances between each contour point and all of its neighboring points is cal- 
culated, transforming Eq. 6.2 into Eq. 8. By consolidating the observations above, 
the generic active contours equation shown in Eq. 6 customized for 3D implemen- 
tation is written as Eq. 9. 

m 

E[on,-T.i^i-\\Pi-M)\) (7) 

./=1 

di = ^ ■'- (7.1) 

m 

m 
ELn.^Y.iP'-^'if (8) 

+ KX-IIV/II (9) 

As we choose to implement active contours via greedy algorithm for its sim- 
plicity and low computational complexity [12], it is crucial to normalize the con- 
tribution of each energy term. Hence, for each contour point p, , surface continuity 
and smoothness terms are normalized by dividing by the corresponding maximum 
value, and surface attraction by the norm of the spatial gradient in the neighborhood 
in which p, moves as shown in Eqs. 10.1, 10.2, 10.3. 



max r' 



norm Tpi cont /in 1\ 



cont 



E 



''"""E' = '-^"^ (10.2) 

ciirv max pi ^ ' 

curv 

_||V/|| -™n£f 

""'■'" £L^ = . ^^ (10.3) 

""s max 17' minp' 

img img 

While we continue to use the same parameter settings of a —I, fi — I and y — 
0.75 after thorough evaluation against the entire dataset, to prevent the algorithm 
from entering an endless loop, two constraints are put in place. First, the number 
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(a) (b) 

Fig. 4 Illustration of (a) 0%, (b) 50%, and (c) 100% congruency 



of iterations / is set to no more than 100. Next, the percentage change in overall 
average distance between contour points Ad must be less than or equal to 0.1% as 
shown in Eqs. 11, 11.1 such that Va — 1 . . . I . These constraints are also chosen after 
evaluating the method against the entire dataset. 
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(11) 
(11.1) 



4 Implementation and Results 
4.1 Evaluation Criteria 



To ensure validity and comparability throughout our studies, we develop a measure 
of congruency expressed in percentages as shown in Eq. 12 where V^ is the delin- 
eated volume and Vj, is the base comparative volume - 0% means no match whereas 
100% represent a perfect match. See Fig. 4 for illustration. 



C- 



(12) 



The congruency coefficient is performed for each image pair (i.e., the base and 
the delineated), and the corresponding /x and /x ± 2cr values for the entire set are 
then calculated for visualization via Bland-Altman graphs when desired. Given our 
validation is mainly based on the usage of mean and standard deviation of a dataset's 
level of agreement measured in percentages, the results are presented in the form of 
"mean: standard deviation", e.g., 100:0% means perfect agreement. 



4.2 Simulation Datasets 



We first confirm the robustness and accuracy of our methods against the three 
known-volume simulation datasets previously mentioned: (1) 90 simulations with 
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(a) 


SET1 


SET 2 


SET 3 


Method A Method B Method C Method D Method E 


Method A Method C Method D Method E 


Method A Method D Method E 


p-2o 


31.79% 72.76% 67.08% 71.98% 60.64% 


92.48% 87.29% 91.54% 92.07% 


92.34% 92.62% 89.29% 


M 


78.00% 88.19% 85.08% 89.83% 81.81% 


97.44% 93.83% 96.16% 96.99% 


97.37% 96.93% 95.60% 


(J 


23.11% 7.71% 9.00% 8.92% 10.58% 


2.48% 3.27% 2.31% 2.46% 


2.51% 2.15% 3.16% 


\i*2a 


124.21% 103.61% 103.07% 107.67% 102.97% 


102.40% 100.36% 100.78% 101.91% 


102.39% 101.23% 101.91% 



(b) 




SET 4 






SETS 




SET 6 


Method B 


Method C 


Method 


Method A 


Method C 


Method D 


Method A 


Method D 


(1-20 


84.45% 


90.24% 


90.96% 


91.27% 


85.43% 


87.40% 


90.87% 


85.95% 


M 


95.00% 


93.62% 


95.15% 


96.34% 


90.88% 


92.85% 


96.37% 


92.33% 





5.27% 


1.69% 


2.09% 


2.53% 


2.72% 


2.73% 


2.75% 


3.19% 


\i*2a 


105.64% 


97.00% 


99.33% 


101.41% 


96.32% 


98.31% 


101.87% 


98.70% 



KEY 

SET 1 : 90 simulations with low MCV/TCV 
SET 2: 50 simulations with normal MCV/TCV 
SET 3; 30 simulations with hotspots 
SET 4: 35 subject scans with low MCV/TCV 
SET 5: 50 subject scans with normal MCV/TCV 
SET 6: 25 subject scans with hotspots 



Method A: DUET 

Method B: DUET with planar snake 
Method C: DUET with partial 3D snake 
Method D: DUET with full 3D snake 
Method E: Experienced physicians 



Fig. 5 Volumetric agreement results in percentages for (a) three sets of simulations, and (b) three 
sets of real subject SPECT scans 



low MCV/TCV, (2) 50 simulations with normal MCV/TCV, and (3) 30 simulations 
with high MCV due to artificially inserted hotspots. Using the known phantom vol- 
ume of 61,660 voxels as base volume, congruency is calculated with the delineated 
volume being the volume extracted by each of our methods. For the set of 50 normal 
simulations, near-perfect average agreements of 96:2% to 97:3% is obtained across 
our methods. With the set of 90 low MCV/TCV simulations, an average agreement 
of 90:9% is achieved using 3D active contours - a significant improvement over 
78:23% and 88:8% achieved by DUET alone and planar active contours respec- 
tively. Finally, with the set of 30 hotspot-infected simulations, 3D active contours 
maintain approximately similar mean congruency of 97:2% as DUET alone. See 
Fig. 5a for details. 



4.3 Real Subject Datasets 



We now evaluate against the three sets of real subject SPECT scans using results 
obtained by experienced physicians as the gold standard: (1) 35 scans with low 
MCV/TCV, (2) 50 scans with normal MCV/TCV, and (3) 25 scans with high MCV 
due to hotspots. Using results obtained by experienced physicians as the compara- 
tive base, average agreements of 95:2% are achieved by our methods for the set of 
50 normal scans. For the set of 35 low-count SPECT scans, while we achieve 93:3% 
mean agreement via 3D active contours - a marginal improvement over previous at- 
tempts at planar/partial-3D active contours - performance degradation is observed 
when comparing against the 96:3% achieved by DUET alone. Finally, with the set 
of 25 hotspot-infected scans, we achieve yet another highly satisfactory agreement 
of 92:3% on average. See Fig. 5b for details. 
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(a) 



(b) 






(c) 



Fig. 6 Sample delineation of a hotspot-infected subject SPECT scan using (a) DUET alone and 
(b) 3D active contours, with the difference between the two methods shown in (c) 

5 Discussion 
5.1 Findings 

Through our studies, we have shown our success in developing and implementing 
a SPECT lung delineation method that is truly 3D in nature, achieving an overall 
agreement of 90% for both simulations and real subject SPECT scans. With low- 
count SPECT scans, while 3D active contours achieve similar mean agreement of 
95% in comparison to the traditional planar approach, it has a lower variance of 
2% instead of 5%. It has also improved upon the limitation of over delineation evi- 
dent in our previous partial attempt at 3D active contours with a marginal increase 
of 1.5:0.5% in agreement. The results are also consistent with those obtained from 
simulations with low MCV/TCV. For normally ventilated scans and simulations, the 
high average agreements of 96:3% and 97:2.5% indicate that DUET alone is suffi- 
cient for delineation. Finally, with hotspot-infected subject scans, 3D active contours 
maintain its ability to accurately delineate lung volumes up to 92:3% mean agree- 
ment. While it is evident that 3D active contours yield excellent results in general, 
it is proven to be especially useful in dealing with low-count scans. Furthermore, 
due to lack of information on the actual lung volumes of real subject SPECT scans, 
closer examination of the delineated lung contours of normal and hotspot-infected 
simulations reveal 3D active contours to be slightly over-aggressive in removing 
some true peripheral lung tissues. See Fig. 6 for illustration. 



5.2 Future Work 

Through our progressive studies we have shown the successful development and 
implementation of a SPECT lung delineation method that is truly 3D in nature. 
Although additional work is still required to identify the true cause of the over- 
aggressiveness of the method, examination of Fig. 6 reveal that the method attracts 
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the contour towards the hotspot edges with higher gradient difference. Furthermore, 
smaller regions with less prominent voxel values are excluded due to current limita- 
tion in segmenting multiple objects. Hence, immediate improvements can be made 
by incorporating segmentation of multiple objects, and reducing the sensitivity of 
the method towards hotspot edges. Furthermore, dynamic adjustment of the param- 
eters in active contours may be developed and/or additional energy term(s) intro- 
duced. From a dataset perspective, although we have developed fast methods for 
mass generation of simulations and hotspots, additional phantoms that contain de- 
fect(s) similar to those seen in subject scans induced by hotspots and/or other car- 
diopulmonary disorders are also desired. 



6 Conclusion 

Although SPECT-CT hybrid machines have been introduced with supporting stud- 
ies performed on extracting lung contours using anatomical information provided 
by CT, high total costs is still a major factor that impedes their installation and us- 
age on a wide scale [13, 14]. Based on this notion, we have successfully developed 
a hybrid method of dynamic thresholding and 3D active contours in delineating 
SPECT lung contours for PE diagnosis, and the detection of other cardiopulmonary 
disorders. While the results reveal marginal delineation of true peripheral lung tis- 
sues evaluation for scans with normal to high MCV/TCV, overall congruency is still 
above 90%. The method is most useful in delineating low-count scans with 95% 
agreement on average. All in all, we have successfully developed a SPECT lung 
delineation method that is truly 3D in all aspects. 

Acknowledgements We would like to thank the physicians and staff at the Royal North Shore 
Hospital, Australia for the provisioning of the original datasets as well as medical guidance 
throughout our studies. 
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